> ## Documentation Index
> Fetch the complete documentation index at: https://developer.watson-orchestrate.ibm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# wxO CI/CD Deployment Approach (Part 2)

> Configuration specifications, knowledge-base deployment considerations, and secret handling patterns for wxO CI/CD workflows

## 5) Configuration Specifications (Per Branch)

### 5.1 Tenant Configuration

Each branch contains `config/tenant.yaml`, which specifies which wxO tenant and environment to target.

<Tabs>
  <Tab title="Dev">
    **`dev` branch — `config/tenant.yaml`** (`dev-qa-tenant`, Draft)

    ```yaml theme={null}
    tenant:
      name: dev-qa-tenant
      id: "tenant-12345-dev-qa"
      wxo_environment: draft        # wxO's Draft environment
      api_endpoint: "https://dev-qa.wxo.example.com/api"
      auth:
        type: api_key
        secret_ref: "wxo/dev-qa-tenant/api-key"  # reference to secrets manager
    branch: dev
    logical_environment: dev
    ```
  </Tab>

  <Tab title="QA">
    **`qa` branch — `config/tenant.yaml`** (`dev-qa-tenant`, Live)

    ```yaml theme={null}
    tenant:
      name: dev-qa-tenant
      id: "tenant-12345-dev-qa"
      wxo_environment: live         # wxO's Live environment
      api_endpoint: "https://dev-qa.wxo.example.com/api"
      auth:
        type: api_key
        secret_ref: "wxo/dev-qa-tenant/api-key"
    branch: qa
    logical_environment: qa
    ```
  </Tab>

  <Tab title="PreProd">
    **`preprod` branch — `config/tenant.yaml`** (`prod-tenant`, Draft)

    ```yaml theme={null}
    tenant:
      name: prod-tenant
      id: "tenant-67890-prod"
      wxo_environment: draft        # wxO's Draft environment (PreProd)
      api_endpoint: "https://prod.wxo.example.com/api"
      auth:
        type: api_key
        secret_ref: "wxo/prod-tenant/api-key"
    branch: preprod
    logical_environment: preprod
    ```
  </Tab>

  <Tab title="Prod">
    **`prod` branch — `config/tenant.yaml`** (`prod-tenant`, Live)

    ```yaml theme={null}
    tenant:
      name: prod-tenant
      id: "tenant-67890-prod"
      wxo_environment: live         # wxO's Live environment (Prod)
      api_endpoint: "https://prod.wxo.example.com/api"
      auth:
        type: api_key
        secret_ref: "wxo/prod-tenant/api-key"
    branch: prod
    logical_environment: prod
    ```
  </Tab>
</Tabs>

### 5.2 Environment Values

Each branch contains environment-specific configuration in `config/values.yaml`.

<Tabs>
  <Tab title="Dev values">
    **`dev` branch — `config/values.yaml`**

    ```yaml theme={null}
    environment: dev
    tenant: dev-qa-tenant
    featureFlags:
      enableHrToolV2: true
      enableBetaFeatures: true
    endpoints:
      hr_api_base: "https://api.dev.company.com"
      db_host: "db.dev.company.com"
    timeouts:
      api_timeout: 30
      db_timeout: 10
    ```
  </Tab>

  <Tab title="Prod values">
    **`prod` branch — `config/values.yaml`**

    ```yaml theme={null}
    environment: prod
    tenant: prod-tenant
    featureFlags:
      enableHrToolV2: true
      enableBetaFeatures: false
    endpoints:
      hr_api_base: "https://api.prod.company.com"
      db_host: "db.prod.company.com"
    timeouts:
      api_timeout: 60
      db_timeout: 30
    ```
  </Tab>
</Tabs>

### 5.3 Agent Structure (Example)

**`agents/employee_onboarding/agent.yaml`** (evolves per branch)

```yaml theme={null}
spec_version: v1
kind: native
name: employee_onboarding
description: "Automates employee onboarding process"
instructions: |
  You are an HR onboarding assistant that helps automate the employee onboarding process.
  
  Your responsibilities:
  - Create employee records in the HR system
  - Set up accounts and access permissions
  - Schedule onboarding meetings
  - Track onboarding progress
  
  Tool Usage Guidelines:
  1. Before calling a tool, ensure you have all required parameters
  2. Do NOT assume any tool parameters
  3. Do NOT pass parameters that are not explicitly defined
  4. Do NOT call the same tool multiple times with the same parameters
  
  How to Use Tools:
  - Use hr_api to create and update employee records
  - Use db_query to retrieve employee information
  - Use document_processing workflow for document handling

llm: groq/openai/gpt-oss-120b
style: default
tools:
  - hr_api
  - db_query
  - document_processing
knowledge_base:
  - hr_policy_docs
collaborators: []
```

#### Alternative: Minimal Agent Structure

```yaml theme={null}
spec_version: v1
kind: native
name: employee_onboarding
description: "Automates employee onboarding process"
instructions: "You are an HR onboarding assistant."
llm: groq/openai/gpt-oss-120b
style: default
tools:
  - hr_api
  - db_query
collaborators: []
```

<Note>
  * Agents can be created through the **UI or YAML**
  * If agents are built through the UI, they should be **exported as YAML** and versioned in Git
  * If flows are built through the UI, they should be **exported as JSON** and versioned in Git
  * The YAML or JSON files in Git are the **source of truth** for version control
  * The agent structure can differ significantly between branches, for example when `dev` includes experimental tools
</Note>

### Key Agent Fields

* `spec_version`: always `v1`
* `kind`: `native` for wxO native agents, `external` for external agents
* `name`: unique identifier for the agent
* `description`: brief description of the agent's purpose
* `instructions`: natural-language guidance for the LLM
* `llm`: model in the format `provider/developer/model_id`, for example `groq/openai/gpt-oss-120b`
* `style`: agent style such as `default` or `react`
* `tools`: list of tool names the agent can use
* `knowledge_base`: list of knowledge bases. **Currently, only one knowledge base is supported.**
* `collaborators`: list of other agents this agent can call

<Info>
  **Note on credentials:** Agents do **not** have credentials. Credentials such as connection IDs or `app_id` values are associated with **tools only**. When a tool is imported through the CLI, it specifies its required connection ID. Agents use tools, and the tools handle their own authentication.
</Info>

### Agent Deployment Workflow

1. Create the agent YAML file manually
2. List required tools in the agent's `tools:` section
3. Store the agent YAML in Git for version control
4. Deploy flows separately if needed through `orchestrate tools import -k flow`
5. Deploy the agent through `orchestrate agents import -f agent.yaml`
   * Tools listed in `tools:` are automatically deployed
   * Each tool specifies its own connection ID or `app_id` during import
6. Test the agent in the Draft environment
7. Promote to the Live environment when ready through `orchestrate agents deploy`

## 5.4 Knowledge-Base Deployment Considerations

Knowledge bases in wxO have unique deployment characteristics that differ from agents and tools.

### Tenant-Scoped, Not Environment-Scoped

Unlike agents, which have separate Draft and Live states, **knowledge bases are tenant-scoped**:

* A knowledge-base update applies to **both Draft and Live environments** simultaneously
* There is no concept of promoting a knowledge base from Draft to Live within a tenant
* Testing must occur in `dev-qa-tenant` before deployment to `prod-tenant`

### Deployment Order

Knowledge bases must be deployed **before** agents that reference them:

```bash theme={null}
# 1. Deploy knowledge-base first
orchestrate knowledge-bases import -f knowledge-bases/hr_policy_docs.yaml

# 2. Then deploy agent that uses it
orchestrate agents import -f agents/employee_onboarding/agent.yaml
```

### Cross-Tenant Promotion

Knowledge bases must be **explicitly deployed to each tenant**:

* Merging code from `qa-branch` to `staging-branch` does not automatically deploy the knowledge base to `prod-tenant`
* The CI/CD pipeline must deploy knowledge bases to `prod-tenant` separately
* Each tenant maintains its own knowledge-base instance

### Testing Strategy

Because knowledge-base updates affect both Draft and Live:

1. **Test in `dev-qa-tenant` first** by deploying and validating there before `prod-tenant`
2. **Validate queries** in the Draft environment before promoting the agent
3. **Monitor impact** because knowledge-base changes immediately affect Live agents in the same tenant

### Rollback Implications

Rolling back a knowledge base requires redeploying the previous version:

```bash theme={null}
# Checkout previous version
git checkout v1.2.0 -- knowledge-bases/hr_policy_docs.yaml

# Redeploy to tenant
orchestrate knowledge-bases import -f knowledge-bases/hr_policy_docs.yaml
```

<Warning>
  Rolling back an agent does **not** automatically roll back its knowledge base.
</Warning>

### Tool Access

* Credentials are associated with **tools**, not agents
* Each tool specifies its connection ID or `app_id` when imported
* Tools handle their own authentication by using the specified connection
* Agents reference tools by name in the `tools:` section

## 6) Secrets Handling (No Secrets in Git)

* Use the **customer's secrets manager**, such as Vault, AWS Secrets Manager, Azure Key Vault, or IBM Secrets Manager / Key Protect
* Jenkins fetches secrets at deploy time, exports them as environment variables, and performs substitution
* **Tenant-scoped secrets** ensure complete isolation because each tenant has its own set of secrets

### Secret Organization by Tenant

**Secrets manager structure:**

```text theme={null}
wxo/
  dev-qa-tenant/
    api-key                    # wxO API authentication for dev-qa tenant
    connections/
      hr/username
      hr/password
      hr/client_id
      db/user
      db/pass
      db/host
  prod-tenant/
    api-key                    # wxO API authentication for prod tenant
    connections/
      hr/username
      hr/password
      hr/client_id
      db/user
      db/pass
      db/host
```

### Example Substitution (Bash)

<Tabs>
  <Tab title="Dev environment">
    **For Dev (`dev-qa-tenant`):**

    ```bash theme={null}
    # Fetch tenant-specific wxO API credentials
    export WXO_API_KEY=$(vault kv get -field=api_key secret/wxo/dev-qa-tenant/api-key)
    export WXO_API_ENDPOINT="https://dev-qa.wxo.example.com/api"

    # Fetch connection secrets for dev-qa tenant
    export HR_USERNAME=$(vault kv get -field=username secret/wxo/dev-qa-tenant/connections/hr)
    export HR_PASSWORD=$(vault kv get -field=password secret/wxo/dev-qa-tenant/connections/hr)
    export HR_CLIENT_ID=$(vault kv get -field=client_id secret/wxo/dev-qa-tenant/connections/hr)
    export DB_USER=$(vault kv get -field=user secret/wxo/dev-qa-tenant/connections/db)
    export DB_PASS=$(vault kv get -field=pass secret/wxo/dev-qa-tenant/connections/db)
    export DB_HOST=$(vault kv get -field=host secret/wxo/dev-qa-tenant/connections/db)
    export ENV=dev
    export TENANT=dev-qa-tenant

    # Resolve connections template for dev-qa tenant
    envsubst < connections/dev-qa-tenant/connections.template.yaml > connections/connections-resolved.yaml
    ```
  </Tab>

  <Tab title="Prod environment">
    **For Prod (`prod-tenant`):**

    ```bash theme={null}
    # Fetch tenant-specific wxO API credentials
    export WXO_API_KEY=$(vault kv get -field=api_key secret/wxo/prod-tenant/api-key)
    export WXO_API_ENDPOINT="https://prod.wxo.example.com/api"

    # Fetch connection secrets for prod tenant
    export HR_USERNAME=$(vault kv get -field=username secret/wxo/prod-tenant/connections/hr)
    export HR_PASSWORD=$(vault kv get -field=password secret/wxo/prod-tenant/connections/hr)
    export HR_CLIENT_ID=$(vault kv get -field=client_id secret/wxo/prod-tenant/connections/hr)
    export DB_USER=$(vault kv get -field=user secret/wxo/prod-tenant/connections/db)
    export DB_PASS=$(vault kv get -field=pass secret/wxo/prod-tenant/connections/db)
    export DB_HOST=$(vault kv get -field=host secret/wxo/prod-tenant/connections/db)
    export ENV=prod
    export TENANT=prod-tenant

    # Resolve connections template for prod tenant
    envsubst < connections/prod-tenant/connections.template.yaml > connections/connections-resolved.yaml
    ```
  </Tab>
</Tabs>

<Info>
  **Principles:**

  * Secrets are materialized only inside the pipeline execution context and are never committed back to Git
  * Each tenant has completely isolated secrets to prevent cross-tenant credential leakage
  * Production secrets are never accessible from non-production pipelines
</Info>