> ## Documentation Index
> Fetch the complete documentation index at: https://developer.watson-orchestrate.ibm.com/llms.txt
> Use this file to discover all available pages before exploring further.
# wxO CI/CD Deployment Approach
> A GitOps-inspired deployment strategy for watsonx Orchestrate agents, flows, tools, and knowledge bases using standard CI/CD pipelines
**Use case:** This guide provides **one approach** for customers on **watsonx Orchestrate SaaS** to leverage **Git-based deployment and CI/CD practices** that follow **GitOps principles** to control the lifecycle of agents and associated tools, flows, and other wxO assets.
**Note on CI/CD Neutrality:** There are many valid CI/CD approaches for managing wxO assets. This document presents one specific implementation using Git-based workflows. watsonx Orchestrate maintains a **CI/CD-neutral stance**, providing CLI tools and APIs that enable customers to adopt agent lifecycle management into their own existing CI/CD pipelines, regardless of the specific tools or practices they use (Jenkins, GitLab CI, GitHub Actions, Azure DevOps, and others).
**Scope:** While this guide focuses on **agents, flows, knowledge, and tools**, the same approach can be applied to other wxO features that have file-based assets, including but not limited to:
* **Voice** (voice configurations)
* **Custom Models** (custom LLM model configurations)
* **Plugins** (plugin definitions and configurations)
Each of these features has corresponding CLI commands and file-based representations that can follow the same GitOps workflow outlined in this document.
## Executive Summary
This document proposes a **GitOps-inspired deployment strategy** for watsonx Orchestrate (wxO) agents, flows, and tools that works without Kubernetes or Argo CD access. The approach uses standard CI/CD tools such as Jenkins, GitLab CI, or GitHub Actions together with the wxO CLI.
### Key Components
* **Tenant 1 (`dev-qa-tenant`)**: Draft → Dev, Live → QA
* **Tenant 2 (`prod-tenant`)**: Draft → PreProd, Live → Prod
* Provides complete isolation between production and non-production environments
* Git branches represent environment state (`dev`, `qa`, `preprod`, `prod`)
* Git tags mark deployments (`v1.2.3-dev`, `v1.2.3-qa`, and so on)
* Git merges promote code between environments
* No manual version directories; Git handles all versioning
* **Agents**: Deployed via the wxO CLI (`orchestrate agents import` or `orchestrate agents deploy`)
* Agent YAML files are hand-written and not CLI-generated
* Tools are automatically deployed when the agent references them in the `tools:` section
* Knowledge bases are automatically deployed when the agent references them in the `knowledge_base:` section. **One agent can only have one knowledge base.**
* **Flows**: Can be deployed separately through `orchestrate tools import -k flow`
* **Knowledge bases**: Deployed through `orchestrate knowledge-bases import`
* **Tenant-scoped** rather than environment-scoped, so updates apply to both Draft and Live simultaneously
* Must be deployed before agents that reference them
* See [Creating knowledge bases](/knowledge_base/build_kb) for more information
* **Secrets**: Never stored in Git; resolved from a secret manager at deploy time
* **Promotion**: Managed through Git branch merges with pull request approvals
* Agents can be managed in separate Git repositories
* The CI/CD pipeline is the only authorized deployment path
* Dependencies must be coordinated through deployment order and testing
### Deployment Flow
```text theme={null}
Code Change → Git Branch → CI/CD Pipeline → wxO Deployment
develop → dev-qa-tenant Draft → Auto-deploy
↓ (PR: develop → qa-)
qa- → dev-qa-tenant Live → Auto-deploy + approval
↓ (PR: qa- → staging-) ← CROSS-TENANT BOUNDARY
staging- → prod-tenant Draft → Auto-deploy + approval
↓ (PR: staging- → main)
main → prod-tenant Live → Auto-deploy + CAB approval
```
### Key Benefits
Works without requiring Kubernetes access
Separates production from non-production environments
Uses Git tags for release tracking
Uses Git merges with approval gates
Supports rollback through `git revert`
Coordinates deployments across repositories
Resolves secrets at runtime
Preserves deployment history in Git
### Document Structure
This guide is organized into four parts:
Goals, two-tenant architecture, repository layout, and Git-based versioning
Configuration specifications, knowledge-base deployment, and secret handling
Inter-agent dependencies, testing gates, promotion workflows, and CLI deployment
Core principles summary and implementation checklist
## Important Disclaimer
**All code examples, configurations, and scripts in this document are provided for reference purposes only.**
These examples are intended to illustrate concepts and approaches. They **must be customized and adapted** to match your organization's specific:
* Deployment practices and standards
* Security requirements and policies
* CI/CD tooling and infrastructure
* Network architecture and access controls
* Compliance and governance requirements
* Organizational workflows and approval processes
**Before implementing:**
* Review all examples with your security, DevOps, and compliance teams
* Adapt code to your organization's coding standards and best practices
* Test thoroughly in non-production environments
* Validate against your organization's security policies
* Ensure compliance with relevant regulations and standards
**This document does not constitute:**
* Production-ready code or configurations
* Security-hardened implementations
* Officially supported IBM solutions or recommendations
* A substitute for proper security review and testing
## 1) Goals and Non-Goals
### Goals
* Deterministic, auditable promotion of **agents** (YAML), **flows** (Python or JSON), and **tools** (Python) across **Dev → QA → PreProd → Prod**
* Support multiple versions of the same agent; flows and tools are **shared** rather than tied one-to-one to agents
* **No credentials in Git**; secrets are resolved at deploy time from a secret manager
* Safe rollbacks, easy diffs, and clear separation of concerns
### Non-Goals
* Does not require Kubernetes or Argo CD access for customers
* Does not mandate a single secrets backend; compatible with Vault, AWS, Azure, IBM, and similar secret managers
## 2) Two-Tenant Architecture
### Current wxO environments per tenant
wxO currently provides **only two environments per tenant**:
* **Draft** for development and testing
* **Live** for production use
### Proposed Solution: Two Tenants
To support a full **Dev → QA → PreProd → Prod** promotion pipeline, use **two separate wxO tenants or accounts**.
**Non-Production Account (`dev-qa-tenant`)**
This single tenant or account contains both Dev and QA environments:
* **Draft environment** → **Dev** (`develop` branch)
* **Live environment** → **QA** (`qa-` branches)
Dev and QA are in the **same tenant or account**. Dev uses the Draft environment, and QA uses the Live environment.
**Production Account (`prod-tenant`)**
This single tenant or account contains both PreProd and Prod environments:
* **Draft environment** → **PreProd or Staging** (`staging-` branches)
* **Live environment** → **Production** (`main` branch)
PreProd and Prod are in the **same tenant or account**. PreProd uses the Draft environment, and Prod uses the Live environment.
A knowledge base created in a tenant is used for both Draft and Live environments.
### Key Architecture Points
**Environment Mapping:**
```text theme={null}
dev-qa-tenant (Account 1)
├─ Draft = Dev
└─ Live = QA
prod-tenant (Account 2)
├─ Draft = PreProd/Staging
└─ Live = Production
```
**Rationale:**
* **Isolation**: Production is completely isolated from development and testing activities
* **Security**: Different RBAC, credentials, and network policies per account
* **Compliance**: Audit trails and change controls are separated by account
* **Risk mitigation**: Issues in Dev or QA cannot affect PreProd or Prod infrastructure
* **Same-account promotion**: Dev → QA and PreProd → Prod use wxO's native Draft → Live promotion
### Tenant Configuration
Each tenant requires:
* Separate **API endpoints** and **authentication tokens**
* Separate **secrets** in the secret manager, scoped by tenant
* Separate **connection configurations**
* Tenant-specific **feature flags** and **resource quotas**
### Cross-Tenant Promotion
Promoting from **QA → PreProd** involves:
1. Updating version pins in `envs/preprod/*-releases.yaml`
2. The pipeline authenticates to **`prod-tenant`** instead of **`dev-qa-tenant`**
3. The deployment targets the **Draft** environment in **`prod-tenant`** for PreProd
4. After validation, the deployment is promoted to the **Live** environment in **`prod-tenant`** for Prod
## 3) Repository Layout and Branching Strategy
### Git Branch Structure
```text theme={null}
develop (dev-qa-tenant Draft)
↓
qa- (dev-qa-tenant Live)
↓
staging- (prod-tenant Draft)
↓
main (prod-tenant Live) ← PRODUCTION
```
### Branch Convention
* `develop`: current development branch, auto-deploys to `dev-qa-tenant` Draft
* `qa-`: QA release branch, auto-deploys to `dev-qa-tenant` Live
* `staging-`: pre-production branch, auto-deploys to `prod-tenant` Draft
* `main`: production branch, auto-deploys to `prod-tenant` Live
### Branch Protection Rules
* `develop`: requires pull request reviews for feature merges
* `qa-`: created from `develop`, requires approval to promote
* `staging-`: created from the QA branch, requires approval and cross-tenant validation
* `main`: requires CAB approval, change ticket, and final sign-off
### Repository Structure Per Branch
```text theme={null}
repo/
agents/
employee_onboarding/
agent.yaml
document_processing_agent/
agent.yaml
tools/
hr_api/
tool.py
db_query/
tool.py
document_processing/
workflow.py
hr_workflow/
workflow.py
knowledge-bases/
hr_policy_docs.yaml
product_catalog.yaml
connections/
dev-qa-tenant/
connections.template.yaml
prod-tenant/
connections.template.yaml
config/
tenant.yaml
values.yaml
ci/
pipelines/
jenkins/
Jenkinsfile
policies/
opa/rego/...
docs/
adr/
runbooks/
```
### Rationale
* **Git branches** provide versioning, and each branch represents the state of a specific environment
* **No version directories** are needed because agent, tool, and flow files sit at the root of their folders
* **Git tags** mark releases, for example `v1.2.0` or `release-2024-01-15`
* **Branch merging** promotes code from one environment to the next
* **Tenant separation** is defined by the branch-specific `config/tenant.yaml`
* **Secrets** are never stored in Git; only templates and secret references are committed
* **Structural changes** can evolve naturally through branches
* The **`tools/`** directory can contain both Python or OpenAPI tools and agentic workflows
## 4) Git-Based Versioning Strategy
### Branch-Based Versioning
* **Git branches** represent environment state; each branch contains the current version for that environment
* **Git tags** mark releases using **semantic versioning** (`MAJOR.MINOR.PATCH`)
* `MAJOR`: breaking changes in I/O or behavior
* `MINOR`: backward-compatible features
* `PATCH`: bug fixes
* **Git commits** provide a full audit trail of all changes
* **Branch merging** promotes code from one environment to the next
### Backward Compatibility Principle
**Critical design principle:** Agents and tools should be designed to **maintain backward compatibility**. Upgrading an agent or tool should **not break existing functionality** that other agents or systems depend on.
#### Why backward compatibility matters
1. **Single active version**: Because only one version of an agent or tool is active in each environment, breaking changes can immediately affect all dependent agents
2. **Deployment safety**: Backward-compatible changes allow safe, incremental deployments without extra coordination overhead
3. **Reduced risk**: They reduce the chance of cascading failures across dependent agents
4. **Simplified rollback**: Rollback is easier when changes are additive instead of breaking
#### Best practices for backward compatibility
* **Additive changes**: Add new parameters or features without removing or changing existing ones
* **Optional parameters**: New parameters should have sensible defaults
* **Deprecation path**: If breaking changes are unavoidable:
1. Add new functionality alongside the old implementation
2. Mark the old functionality as deprecated with warnings
3. Remove the old functionality only in the next major version
* **Tool I/O contracts**: Maintain consistent input and output schemas; extend rather than modify
* **Agent instructions**: Update prompts to support new features while preserving existing behavior
* **Version communication**: Clearly document breaking changes in release notes and Git tags
#### When breaking changes are necessary
Because wxO supports **only one agent version per environment**, it is better to **create a new agent with a different name** rather than upgrade an existing agent in a breaking way.
#### Recommended approach for breaking changes
1. **Create a new agent** such as `employee_onboarding_v2` or `employee_onboarding_enhanced`
* Give the new agent a distinct name
* Allow both old and new agents to coexist in the same environment
* Let dependent systems migrate at their own pace
2. **Maintain the old agent**
* Keep the original agent available for existing consumers
* Mark it as deprecated in documentation
* Decommission it after all consumers migrate
3. **Coordinate migration**
* Update dependent agents or systems to use the new agent name
* Test thoroughly in lower environments
* Migrate production consumers in a controlled way
* Monitor both agents during the transition
4. **Decommission the old agent**
* Remove it after all consumers have migrated
* Document the migration in release notes
### Why this approach works better
Both agents can run during migration
Consumers can move at their own pace
Consumers can switch back if needed
Distinct names make versions obvious
### Example: Tool Evolution (Backward Compatible)
```python theme={null}
# v1.0.0 - Original tool
def process_document(document_path):
return {"status": "processed", "path": document_path}
# v1.1.0 - Backward compatible (adds optional parameter)
def process_document(document_path, format="pdf"):
# format defaults to "pdf" - existing callers unaffected
return {"status": "processed", "path": document_path, "format": format}
# v1.2.0 - Backward compatible (adds new output field)
def process_document(document_path, format="pdf"):
return {
"status": "processed",
"path": document_path,
"format": format,
"page_count": 10 # New field - existing consumers can ignore
}
```
### Example: Agent Evolution (Breaking Change → New Agent)
```yaml theme={null}
# Original agent: employee_onboarding
# agents/employee_onboarding/agent.yaml
spec_version: v1
kind: native
name: employee_onboarding
llm: groq/openai/gpt-oss-120b
tools:
- hr_api
- document_processor
# New agent with breaking changes: employee_onboarding_v2
# agents/employee_onboarding_v2/agent.yaml
spec_version: v1
kind: native
name: employee_onboarding_v2
llm: groq/openai/gpt-oss-120b
tools:
- hr_api_v2
- document_processor_enhanced
- compliance_checker
```
### Migration Path
1. Deploy `employee_onboarding_v2` alongside `employee_onboarding`
2. Update consuming systems to call `employee_onboarding_v2`
3. Monitor both agents during transition
4. Once all consumers have migrated, decommission `employee_onboarding`
### Tagging Strategy
**Release tags on each branch:**
```bash theme={null}
# Tag format: v{MAJOR}.{MINOR}.{PATCH}-{environment}
v1.2.0-dev
v1.2.0-qa
v1.2.0-preprod
v1.2.0-prod
```
**Annotated tags for traceability:**
```bash theme={null}
git tag -a v1.2.0-prod -m "Release 1.2.0 to production
- Added new HR onboarding workflow
- Fixed document processing bug
- Updated connection timeout settings
Approved by: CAB-2024-001
Deployed: 2024-01-15T10:00:00Z"
```
### Version Evolution Example
```text theme={null}
develop branch:
└─ v1.2.0-dev (deployed to dev-qa-tenant Draft)
qa-1.1.0 branch:
└─ v1.1.0-qa (deployed to dev-qa-tenant Live)
staging-1.0.0 branch:
└─ v1.0.0-staging (deployed to prod-tenant Draft)
main branch:
└─ v1.0.0-prod (deployed to prod-tenant Live)
```
### Benefits of Git-Based Versioning
1. **Natural evolution**: agent structures can change significantly between environments
2. **No manual version management**: Git handles versioning automatically
3. **Full history**: every change is tracked with commit messages
4. **Easy comparison**: `git diff qa..preprod` shows exactly what will be promoted
5. **Simplified structure**: no need for multiple version directories
6. **Branch protection**: helps prevent accidental changes to stable environments