wxO CI/CD Deployment Approach (Part 2) - IBM watsonx Orchestrate ADK

5) Configuration Specifications (Per Branch)

5.1 Tenant Configuration

Each branch contains config/tenant.yaml, which specifies which wxO tenant and environment to target.

Dev
QA
PreProd
Prod

dev branch — config/tenant.yaml (dev-qa-tenant, Draft)

tenant:
  name: dev-qa-tenant
  id: "tenant-12345-dev-qa"
  wxo_environment: draft        # wxO's Draft environment
  api_endpoint: "https://dev-qa.wxo.example.com/api"
  auth:
    type: api_key
    secret_ref: "wxo/dev-qa-tenant/api-key"  # reference to secrets manager
branch: dev
logical_environment: dev

qa branch — config/tenant.yaml (dev-qa-tenant, Live)

tenant:
  name: dev-qa-tenant
  id: "tenant-12345-dev-qa"
  wxo_environment: live         # wxO's Live environment
  api_endpoint: "https://dev-qa.wxo.example.com/api"
  auth:
    type: api_key
    secret_ref: "wxo/dev-qa-tenant/api-key"
branch: qa
logical_environment: qa

preprod branch — config/tenant.yaml (prod-tenant, Draft)

tenant:
  name: prod-tenant
  id: "tenant-67890-prod"
  wxo_environment: draft        # wxO's Draft environment (PreProd)
  api_endpoint: "https://prod.wxo.example.com/api"
  auth:
    type: api_key
    secret_ref: "wxo/prod-tenant/api-key"
branch: preprod
logical_environment: preprod

prod branch — config/tenant.yaml (prod-tenant, Live)

tenant:
  name: prod-tenant
  id: "tenant-67890-prod"
  wxo_environment: live         # wxO's Live environment (Prod)
  api_endpoint: "https://prod.wxo.example.com/api"
  auth:
    type: api_key
    secret_ref: "wxo/prod-tenant/api-key"
branch: prod
logical_environment: prod

5.2 Environment Values

Each branch contains environment-specific configuration in config/values.yaml.

Dev values
Prod values

dev branch — config/values.yaml

environment: dev
tenant: dev-qa-tenant
featureFlags:
  enableHrToolV2: true
  enableBetaFeatures: true
endpoints:
  hr_api_base: "https://api.dev.company.com"
  db_host: "db.dev.company.com"
timeouts:
  api_timeout: 30
  db_timeout: 10

prod branch — config/values.yaml

environment: prod
tenant: prod-tenant
featureFlags:
  enableHrToolV2: true
  enableBetaFeatures: false
endpoints:
  hr_api_base: "https://api.prod.company.com"
  db_host: "db.prod.company.com"
timeouts:
  api_timeout: 60
  db_timeout: 30

5.3 Agent Structure (Example)

agents/employee_onboarding/agent.yaml (evolves per branch)

spec_version: v1
kind: native
name: employee_onboarding
description: "Automates employee onboarding process"
instructions: |
  You are an HR onboarding assistant that helps automate the employee onboarding process.
  
  Your responsibilities:
  - Create employee records in the HR system
  - Set up accounts and access permissions
  - Schedule onboarding meetings
  - Track onboarding progress
  
  Tool Usage Guidelines:
  1. Before calling a tool, ensure you have all required parameters
  2. Do NOT assume any tool parameters
  3. Do NOT pass parameters that are not explicitly defined
  4. Do NOT call the same tool multiple times with the same parameters
  
  How to Use Tools:
  - Use hr_api to create and update employee records
  - Use db_query to retrieve employee information
  - Use document_processing workflow for document handling

llm: groq/openai/gpt-oss-120b
style: default
tools:
  - hr_api
  - db_query
  - document_processing
knowledge_base:
  - hr_policy_docs
collaborators: []

Alternative: Minimal Agent Structure

spec_version: v1
kind: native
name: employee_onboarding
description: "Automates employee onboarding process"
instructions: "You are an HR onboarding assistant."
llm: groq/openai/gpt-oss-120b
style: default
tools:
  - hr_api
  - db_query
collaborators: []

Agents can be created through the UI or YAML
If agents are built through the UI, they should be exported as YAML and versioned in Git
If flows are built through the UI, they should be exported as JSON and versioned in Git
The YAML or JSON files in Git are the source of truth for version control
The agent structure can differ significantly between branches, for example when dev includes experimental tools

Key Agent Fields

spec_version: always v1
kind: native for wxO native agents, external for external agents
name: unique identifier for the agent
description: brief description of the agent’s purpose
instructions: natural-language guidance for the LLM
llm: model in the format provider/developer/model_id, for example groq/openai/gpt-oss-120b
style: agent style such as default or react
tools: list of tool names the agent can use
knowledge_base: list of knowledge bases. Currently, only one knowledge base is supported.
collaborators: list of other agents this agent can call

Note on credentials: Agents do not have credentials. Credentials such as connection IDs or app_id values are associated with tools only. When a tool is imported through the CLI, it specifies its required connection ID. Agents use tools, and the tools handle their own authentication.

Agent Deployment Workflow

Create the agent YAML file manually
List required tools in the agent’s tools: section
Store the agent YAML in Git for version control
Deploy flows separately if needed through orchestrate tools import -k flow
Deploy the agent through orchestrate agents import -f agent.yaml
- Tools listed in tools: are automatically deployed
- Each tool specifies its own connection ID or app_id during import
Test the agent in the Draft environment
Promote to the Live environment when ready through orchestrate agents deploy

5.4 Knowledge-Base Deployment Considerations

Knowledge bases in wxO have unique deployment characteristics that differ from agents and tools.

Tenant-Scoped, Not Environment-Scoped

Unlike agents, which have separate Draft and Live states, knowledge bases are tenant-scoped:

A knowledge-base update applies to both Draft and Live environments simultaneously
There is no concept of promoting a knowledge base from Draft to Live within a tenant
Testing must occur in dev-qa-tenant before deployment to prod-tenant

Deployment Order

Knowledge bases must be deployed before agents that reference them:

# 1. Deploy knowledge-base first
orchestrate knowledge-bases import -f knowledge-bases/hr_policy_docs.yaml

# 2. Then deploy agent that uses it
orchestrate agents import -f agents/employee_onboarding/agent.yaml

Cross-Tenant Promotion

Knowledge bases must be explicitly deployed to each tenant:

Merging code from qa-branch to staging-branch does not automatically deploy the knowledge base to prod-tenant
The CI/CD pipeline must deploy knowledge bases to prod-tenant separately
Each tenant maintains its own knowledge-base instance

Testing Strategy

Because knowledge-base updates affect both Draft and Live:

Test in dev-qa-tenant first by deploying and validating there before prod-tenant
Validate queries in the Draft environment before promoting the agent
Monitor impact because knowledge-base changes immediately affect Live agents in the same tenant

Rollback Implications

Rolling back a knowledge base requires redeploying the previous version:

# Checkout previous version
git checkout v1.2.0 -- knowledge-bases/hr_policy_docs.yaml

# Redeploy to tenant
orchestrate knowledge-bases import -f knowledge-bases/hr_policy_docs.yaml

Rolling back an agent does not automatically roll back its knowledge base.

Tool Access

Credentials are associated with tools, not agents
Each tool specifies its connection ID or app_id when imported
Tools handle their own authentication by using the specified connection
Agents reference tools by name in the tools: section

6) Secrets Handling (No Secrets in Git)

Use the customer’s secrets manager, such as Vault, AWS Secrets Manager, Azure Key Vault, or IBM Secrets Manager / Key Protect
Jenkins fetches secrets at deploy time, exports them as environment variables, and performs substitution
Tenant-scoped secrets ensure complete isolation because each tenant has its own set of secrets

Secret Organization by Tenant

Secrets manager structure:

wxo/
  dev-qa-tenant/
    api-key                    # wxO API authentication for dev-qa tenant
    connections/
      hr/username
      hr/password
      hr/client_id
      db/user
      db/pass
      db/host
  prod-tenant/
    api-key                    # wxO API authentication for prod tenant
    connections/
      hr/username
      hr/password
      hr/client_id
      db/user
      db/pass
      db/host

Example Substitution (Bash)

Dev environment
Prod environment

For Dev (dev-qa-tenant):

# Fetch tenant-specific wxO API credentials
export WXO_API_KEY=$(vault kv get -field=api_key secret/wxo/dev-qa-tenant/api-key)
export WXO_API_ENDPOINT="https://dev-qa.wxo.example.com/api"

# Fetch connection secrets for dev-qa tenant
export HR_USERNAME=$(vault kv get -field=username secret/wxo/dev-qa-tenant/connections/hr)
export HR_PASSWORD=$(vault kv get -field=password secret/wxo/dev-qa-tenant/connections/hr)
export HR_CLIENT_ID=$(vault kv get -field=client_id secret/wxo/dev-qa-tenant/connections/hr)
export DB_USER=$(vault kv get -field=user secret/wxo/dev-qa-tenant/connections/db)
export DB_PASS=$(vault kv get -field=pass secret/wxo/dev-qa-tenant/connections/db)
export DB_HOST=$(vault kv get -field=host secret/wxo/dev-qa-tenant/connections/db)
export ENV=dev
export TENANT=dev-qa-tenant

# Resolve connections template for dev-qa tenant
envsubst < connections/dev-qa-tenant/connections.template.yaml > connections/connections-resolved.yaml

For Prod (prod-tenant):

# Fetch tenant-specific wxO API credentials
export WXO_API_KEY=$(vault kv get -field=api_key secret/wxo/prod-tenant/api-key)
export WXO_API_ENDPOINT="https://prod.wxo.example.com/api"

# Fetch connection secrets for prod tenant
export HR_USERNAME=$(vault kv get -field=username secret/wxo/prod-tenant/connections/hr)
export HR_PASSWORD=$(vault kv get -field=password secret/wxo/prod-tenant/connections/hr)
export HR_CLIENT_ID=$(vault kv get -field=client_id secret/wxo/prod-tenant/connections/hr)
export DB_USER=$(vault kv get -field=user secret/wxo/prod-tenant/connections/db)
export DB_PASS=$(vault kv get -field=pass secret/wxo/prod-tenant/connections/db)
export DB_HOST=$(vault kv get -field=host secret/wxo/prod-tenant/connections/db)
export ENV=prod
export TENANT=prod-tenant

# Resolve connections template for prod tenant
envsubst < connections/prod-tenant/connections.template.yaml > connections/connections-resolved.yaml

Principles:

Secrets are materialized only inside the pipeline execution context and are never committed back to Git
Each tenant has completely isolated secrets to prevent cross-tenant credential leakage
Production secrets are never accessible from non-production pipelines

​5) Configuration Specifications (Per Branch)

​5.1 Tenant Configuration

​5.2 Environment Values

​5.3 Agent Structure (Example)

​Alternative: Minimal Agent Structure

​Key Agent Fields

​Agent Deployment Workflow

​5.4 Knowledge-Base Deployment Considerations

​Tenant-Scoped, Not Environment-Scoped

​Deployment Order

​Cross-Tenant Promotion

​Testing Strategy

​Rollback Implications

​Tool Access

​6) Secrets Handling (No Secrets in Git)

​Secret Organization by Tenant

​Example Substitution (Bash)

5) Configuration Specifications (Per Branch)

5.1 Tenant Configuration

5.2 Environment Values

5.3 Agent Structure (Example)

Alternative: Minimal Agent Structure

Key Agent Fields

Agent Deployment Workflow

5.4 Knowledge-Base Deployment Considerations

Tenant-Scoped, Not Environment-Scoped

Deployment Order

Cross-Tenant Promotion

Testing Strategy

Rollback Implications

Tool Access

6) Secrets Handling (No Secrets in Git)

Secret Organization by Tenant

Example Substitution (Bash)