wxO CI/CD Deployment Approach - IBM watsonx Orchestrate ADK

Use case: This guide provides one approach for customers on watsonx Orchestrate SaaS to leverage Git-based deployment and CI/CD practices that follow GitOps principles to control the lifecycle of agents and associated tools, flows, and other wxO assets.Note on CI/CD Neutrality: There are many valid CI/CD approaches for managing wxO assets. This document presents one specific implementation using Git-based workflows. watsonx Orchestrate maintains a CI/CD-neutral stance, providing CLI tools and APIs that enable customers to adopt agent lifecycle management into their own existing CI/CD pipelines, regardless of the specific tools or practices they use (Jenkins, GitLab CI, GitHub Actions, Azure DevOps, and others).

Scope: While this guide focuses on agents, flows, knowledge, and tools, the same approach can be applied to other wxO features that have file-based assets, including but not limited to:

Voice (voice configurations)
Custom Models (custom LLM model configurations)
Plugins (plugin definitions and configurations)

Each of these features has corresponding CLI commands and file-based representations that can follow the same GitOps workflow outlined in this document.

Executive Summary

This document proposes a GitOps-inspired deployment strategy for watsonx Orchestrate (wxO) agents, flows, and tools that works without Kubernetes or Argo CD access. The approach uses standard CI/CD tools such as Jenkins, GitLab CI, or GitHub Actions together with the wxO CLI.

Key Components

1. Two-Tenant Architecture

Tenant 1 (dev-qa-tenant): Draft → Dev, Live → QA
Tenant 2 (prod-tenant): Draft → PreProd, Live → Prod
Provides complete isolation between production and non-production environments

2. Git-Based Versioning

Git branches represent environment state (dev, qa, preprod, prod)
Git tags mark deployments (v1.2.3-dev, v1.2.3-qa, and so on)
Git merges promote code between environments
No manual version directories; Git handles all versioning

3. Deployment Approach

Agents: Deployed via the wxO CLI (orchestrate agents import or orchestrate agents deploy)
- Agent YAML files are hand-written and not CLI-generated
- Tools are automatically deployed when the agent references them in the tools: section
- Knowledge bases are automatically deployed when the agent references them in the knowledge_base: section. One agent can only have one knowledge base.
Flows: Can be deployed separately through orchestrate tools import -k flow
Knowledge bases: Deployed through orchestrate knowledge-bases import
- Tenant-scoped rather than environment-scoped, so updates apply to both Draft and Live simultaneously
- Must be deployed before agents that reference them
- See Creating knowledge bases for more information
Secrets: Never stored in Git; resolved from a secret manager at deploy time
Promotion: Managed through Git branch merges with pull request approvals

4. Multi-Repository Support

Agents can be managed in separate Git repositories
The CI/CD pipeline is the only authorized deployment path
Dependencies must be coordinated through deployment order and testing

Deployment Flow

Code Change → Git Branch → CI/CD Pipeline → wxO Deployment

develop → dev-qa-tenant Draft → Auto-deploy
  ↓ (PR: develop → qa-<version>)
qa-<version> → dev-qa-tenant Live → Auto-deploy + approval
  ↓ (PR: qa-<version> → staging-<version>) ← CROSS-TENANT BOUNDARY
staging-<version> → prod-tenant Draft → Auto-deploy + approval
  ↓ (PR: staging-<version> → main)
main → prod-tenant Live → Auto-deploy + CAB approval

Key Benefits

GitOps Principles

Works without requiring Kubernetes access

Complete Isolation

Separates production from non-production environments

Automatic Versioning

Uses Git tags for release tracking

Safe Promotions

Uses Git merges with approval gates

Easy Rollbacks

Supports rollback through git revert

Multi-Repository Support

Coordinates deployments across repositories

Secret-Free Git

Resolves secrets at runtime

Audit Trail

Preserves deployment history in Git

Document Structure

This guide is organized into four parts:

Part 1: Foundation

Goals, two-tenant architecture, repository layout, and Git-based versioning

Part 2: Configuration

Configuration specifications, knowledge-base deployment, and secret handling

Part 3: Operations

Inter-agent dependencies, testing gates, promotion workflows, and CLI deployment

Part 4: Summary

Core principles summary and implementation checklist

Important Disclaimer

All code examples, configurations, and scripts in this document are provided for reference purposes only.These examples are intended to illustrate concepts and approaches. They must be customized and adapted to match your organization’s specific:

Deployment practices and standards
Security requirements and policies
CI/CD tooling and infrastructure
Network architecture and access controls
Compliance and governance requirements
Organizational workflows and approval processes

Before implementing:

Review all examples with your security, DevOps, and compliance teams
Adapt code to your organization’s coding standards and best practices
Test thoroughly in non-production environments
Validate against your organization’s security policies
Ensure compliance with relevant regulations and standards

This document does not constitute:

Production-ready code or configurations
Security-hardened implementations
Officially supported IBM solutions or recommendations
A substitute for proper security review and testing

1) Goals and Non-Goals

Goals

Deterministic, auditable promotion of agents (YAML), flows (Python or JSON), and tools (Python) across Dev → QA → PreProd → Prod
Support multiple versions of the same agent; flows and tools are shared rather than tied one-to-one to agents
No credentials in Git; secrets are resolved at deploy time from a secret manager
Safe rollbacks, easy diffs, and clear separation of concerns

Non-Goals

Does not require Kubernetes or Argo CD access for customers
Does not mandate a single secrets backend; compatible with Vault, AWS, Azure, IBM, and similar secret managers

2) Two-Tenant Architecture

Current wxO environments per tenant

wxO currently provides only two environments per tenant:

Draft for development and testing
Live for production use

Proposed Solution: Two Tenants

To support a full Dev → QA → PreProd → Prod promotion pipeline, use two separate wxO tenants or accounts.

Tenant 1: Non-Production
Tenant 2: Production

Non-Production Account (dev-qa-tenant)This single tenant or account contains both Dev and QA environments:

Draft environment → Dev (develop branch)
Live environment → QA (qa-<version> branches)

Dev and QA are in the same tenant or account. Dev uses the Draft environment, and QA uses the Live environment.

Production Account (prod-tenant)This single tenant or account contains both PreProd and Prod environments:

Draft environment → PreProd or Staging (staging-<version> branches)
Live environment → Production (main branch)

PreProd and Prod are in the same tenant or account. PreProd uses the Draft environment, and Prod uses the Live environment.

A knowledge base created in a tenant is used for both Draft and Live environments.

Key Architecture Points

Environment Mapping:

dev-qa-tenant (Account 1)
  ├─ Draft  = Dev
  └─ Live   = QA

prod-tenant (Account 2)
  ├─ Draft  = PreProd/Staging
  └─ Live   = Production

Rationale:

Isolation: Production is completely isolated from development and testing activities
Security: Different RBAC, credentials, and network policies per account
Compliance: Audit trails and change controls are separated by account
Risk mitigation: Issues in Dev or QA cannot affect PreProd or Prod infrastructure
Same-account promotion: Dev → QA and PreProd → Prod use wxO’s native Draft → Live promotion

Tenant Configuration

Each tenant requires:

Separate API endpoints and authentication tokens
Separate secrets in the secret manager, scoped by tenant
Separate connection configurations
Tenant-specific feature flags and resource quotas

Cross-Tenant Promotion

Promoting from QA → PreProd involves:

Updating version pins in envs/preprod/*-releases.yaml
The pipeline authenticates to prod-tenant instead of dev-qa-tenant
The deployment targets the Draft environment in prod-tenant for PreProd
After validation, the deployment is promoted to the Live environment in prod-tenant for Prod

3) Repository Layout and Branching Strategy

Git Branch Structure

develop (dev-qa-tenant Draft)
  ↓
qa-<version> (dev-qa-tenant Live)
  ↓
staging-<version> (prod-tenant Draft)
  ↓
main (prod-tenant Live) ← PRODUCTION

Branch Convention

develop: current development branch, auto-deploys to dev-qa-tenant Draft
qa-<version>: QA release branch, auto-deploys to dev-qa-tenant Live
staging-<version>: pre-production branch, auto-deploys to prod-tenant Draft
main: production branch, auto-deploys to prod-tenant Live

Branch Protection Rules

develop: requires pull request reviews for feature merges
qa-<version>: created from develop, requires approval to promote
staging-<version>: created from the QA branch, requires approval and cross-tenant validation
main: requires CAB approval, change ticket, and final sign-off

Repository Structure Per Branch

repo/
  agents/
    employee_onboarding/
      agent.yaml
    document_processing_agent/
      agent.yaml

  tools/
    hr_api/
      tool.py
    db_query/
      tool.py
    document_processing/
      workflow.py
    hr_workflow/
      workflow.py

  knowledge-bases/
    hr_policy_docs.yaml
    product_catalog.yaml

  connections/
    dev-qa-tenant/
      connections.template.yaml
    prod-tenant/
      connections.template.yaml

  config/
    tenant.yaml
    values.yaml

  ci/
    pipelines/
      jenkins/
        Jenkinsfile
    policies/
      opa/rego/...

  docs/
    adr/
    runbooks/

Rationale

Git branches provide versioning, and each branch represents the state of a specific environment
No version directories are needed because agent, tool, and flow files sit at the root of their folders
Git tags mark releases, for example v1.2.0 or release-2024-01-15
Branch merging promotes code from one environment to the next
Tenant separation is defined by the branch-specific config/tenant.yaml
Secrets are never stored in Git; only templates and secret references are committed
Structural changes can evolve naturally through branches
The tools/ directory can contain both Python or OpenAPI tools and agentic workflows

4) Git-Based Versioning Strategy

Branch-Based Versioning

Git branches represent environment state; each branch contains the current version for that environment
Git tags mark releases using semantic versioning (MAJOR.MINOR.PATCH)
- MAJOR: breaking changes in I/O or behavior
- MINOR: backward-compatible features
- PATCH: bug fixes
Git commits provide a full audit trail of all changes
Branch merging promotes code from one environment to the next

Backward Compatibility Principle

Critical design principle: Agents and tools should be designed to maintain backward compatibility. Upgrading an agent or tool should not break existing functionality that other agents or systems depend on.

Why backward compatibility matters

Single active version: Because only one version of an agent or tool is active in each environment, breaking changes can immediately affect all dependent agents
Deployment safety: Backward-compatible changes allow safe, incremental deployments without extra coordination overhead
Reduced risk: They reduce the chance of cascading failures across dependent agents
Simplified rollback: Rollback is easier when changes are additive instead of breaking

Best practices for backward compatibility

Additive changes: Add new parameters or features without removing or changing existing ones
Optional parameters: New parameters should have sensible defaults
Deprecation path: If breaking changes are unavoidable:
1. Add new functionality alongside the old implementation
2. Mark the old functionality as deprecated with warnings
3. Remove the old functionality only in the next major version
Tool I/O contracts: Maintain consistent input and output schemas; extend rather than modify
Agent instructions: Update prompts to support new features while preserving existing behavior
Version communication: Clearly document breaking changes in release notes and Git tags

When breaking changes are necessary

Because wxO supports only one agent version per environment, it is better to create a new agent with a different name rather than upgrade an existing agent in a breaking way.

Recommended approach for breaking changes

Create a new agent such as employee_onboarding_v2 or employee_onboarding_enhanced
- Give the new agent a distinct name
- Allow both old and new agents to coexist in the same environment
- Let dependent systems migrate at their own pace
Maintain the old agent
- Keep the original agent available for existing consumers
- Mark it as deprecated in documentation
- Decommission it after all consumers migrate
Coordinate migration
- Update dependent agents or systems to use the new agent name
- Test thoroughly in lower environments
- Migrate production consumers in a controlled way
- Monitor both agents during the transition
Decommission the old agent
- Remove it after all consumers have migrated
- Document the migration in release notes

Why this approach works better

No Downtime

Both agents can run during migration

Gradual Migration

Consumers can move at their own pace

Easy Rollback

Consumers can switch back if needed

Clear Separation

Distinct names make versions obvious

Example: Tool Evolution (Backward Compatible)

# v1.0.0 - Original tool
def process_document(document_path):
    return {"status": "processed", "path": document_path}

# v1.1.0 - Backward compatible (adds optional parameter)
def process_document(document_path, format="pdf"):
    # format defaults to "pdf" - existing callers unaffected
    return {"status": "processed", "path": document_path, "format": format}

# v1.2.0 - Backward compatible (adds new output field)
def process_document(document_path, format="pdf"):
    return {
        "status": "processed",
        "path": document_path,
        "format": format,
        "page_count": 10  # New field - existing consumers can ignore
    }

Example: Agent Evolution (Breaking Change → New Agent)

# Original agent: employee_onboarding
# agents/employee_onboarding/agent.yaml
spec_version: v1
kind: native
name: employee_onboarding
llm: groq/openai/gpt-oss-120b
tools:
  - hr_api
  - document_processor

# New agent with breaking changes: employee_onboarding_v2
# agents/employee_onboarding_v2/agent.yaml
spec_version: v1
kind: native
name: employee_onboarding_v2
llm: groq/openai/gpt-oss-120b
tools:
  - hr_api_v2
  - document_processor_enhanced
  - compliance_checker

Migration Path

Deploy employee_onboarding_v2 alongside employee_onboarding
Update consuming systems to call employee_onboarding_v2
Monitor both agents during transition
Once all consumers have migrated, decommission employee_onboarding

Tagging Strategy

Release tags on each branch:

# Tag format: v{MAJOR}.{MINOR}.{PATCH}-{environment}
v1.2.0-dev
v1.2.0-qa
v1.2.0-preprod
v1.2.0-prod

Annotated tags for traceability:

git tag -a v1.2.0-prod -m "Release 1.2.0 to production
- Added new HR onboarding workflow
- Fixed document processing bug
- Updated connection timeout settings
Approved by: CAB-2024-001
Deployed: 2024-01-15T10:00:00Z"

Version Evolution Example

develop branch:
  └─ v1.2.0-dev (deployed to dev-qa-tenant Draft)

qa-1.1.0 branch:
  └─ v1.1.0-qa (deployed to dev-qa-tenant Live)

staging-1.0.0 branch:
  └─ v1.0.0-staging (deployed to prod-tenant Draft)

main branch:
  └─ v1.0.0-prod (deployed to prod-tenant Live)

Benefits of Git-Based Versioning

Natural evolution: agent structures can change significantly between environments
No manual version management: Git handles versioning automatically
Full history: every change is tracked with commit messages
Easy comparison: git diff qa..preprod shows exactly what will be promoted
Simplified structure: no need for multiple version directories
Branch protection: helps prevent accidental changes to stable environments

Release Notes

Get Started

Build

Deploy

Analyze

Developer experience

Legal notices

​Executive Summary

​Key Components

​Deployment Flow

​Key Benefits

GitOps Principles

Complete Isolation

Automatic Versioning

Safe Promotions

Easy Rollbacks

Multi-Repository Support

Secret-Free Git

Audit Trail

​Document Structure

Part 1: Foundation

Part 2: Configuration

Part 3: Operations

Part 4: Summary

​Important Disclaimer

​1) Goals and Non-Goals

​Goals

​Non-Goals

​2) Two-Tenant Architecture

​Current wxO environments per tenant

​Proposed Solution: Two Tenants

​Key Architecture Points

​Tenant Configuration

​Cross-Tenant Promotion

​3) Repository Layout and Branching Strategy

​Git Branch Structure

​Branch Convention

​Branch Protection Rules

​Repository Structure Per Branch

​Rationale

​4) Git-Based Versioning Strategy

​Branch-Based Versioning

​Backward Compatibility Principle

​Why backward compatibility matters

​Best practices for backward compatibility

​When breaking changes are necessary

​Recommended approach for breaking changes

​Why this approach works better

No Downtime

Gradual Migration

Easy Rollback

Clear Separation

​Example: Tool Evolution (Backward Compatible)

​Example: Agent Evolution (Breaking Change → New Agent)

​Migration Path

​Tagging Strategy

​Version Evolution Example

​Benefits of Git-Based Versioning

Executive Summary

Key Components

Deployment Flow

Key Benefits

Document Structure

Important Disclaimer

1) Goals and Non-Goals

Goals

Non-Goals

2) Two-Tenant Architecture

Current wxO environments per tenant

Proposed Solution: Two Tenants

Key Architecture Points

Tenant Configuration

Cross-Tenant Promotion

3) Repository Layout and Branching Strategy

Git Branch Structure

Branch Convention

Branch Protection Rules

Repository Structure Per Branch

Rationale

4) Git-Based Versioning Strategy

Branch-Based Versioning

Backward Compatibility Principle

Why backward compatibility matters

Best practices for backward compatibility

When breaking changes are necessary

Recommended approach for breaking changes

Why this approach works better

Example: Tool Evolution (Backward Compatible)

Example: Agent Evolution (Breaking Change → New Agent)

Migration Path

Tagging Strategy

Version Evolution Example

Benefits of Git-Based Versioning