wxO CI/CD Deployment Approach (Part 3) - IBM watsonx Orchestrate ADK

7) Inter-Agent Dependencies (Multi-Repository)

Overview

When agents are managed in separate Git repositories and have dependencies on each other, such as AgentA depending on AgentB, you need a strategy to coordinate deployments across repositories.

Core Principle: CI/CD Pipeline as the Only Authorized Deployment Path

The general approach is to use the CI/CD pipeline as the only authorized path to deploy agents to production environments. Customers should not deploy agents manually by hand or with ad hoc scripts. This ensures:

Consistency: all deployments follow the same process
Auditability: every deployment is tracked in CI/CD logs
Governance: approval gates and quality checks are enforced
Traceability: deployments are linked to Git commits and pull requests

Key Challenges

Version synchronization: dependent agents must reference compatible versions
Deployment order: dependencies must be deployed before dependents
Cross-tenant coordination: agents must exist in the same tenant and environment
Testing: integration tests require all dependent agents to be available
Single active version: only one version of an agent or tool is active in a given environment

Illustrative Solution: File-Based Agent Registry

The file-based registry approach described below is one possible solution for illustration purposes only. It is not a recommendation or best practice. The final solution might be more sophisticated or simpler depending on the level of lifecycle governance required by your organization.

Given that only one version of an agent or tool is active in a given environment, a simple file-based agent registry can be used to maintain dependencies and validate deployments.

Example Registry Structure

# agent-registry.yaml - File-based catalog for multi-repo dependencies
# This is an ILLUSTRATIVE example, not a prescriptive solution
agents:
  document_processor:
    repository: git@github.com:company/repo-b.git
    dependencies: []  # No dependencies - deploys first
    environments:
      dev: { branch: dev, version: v1.2.3, status: configured }
  
  employee_onboarding:
    repository: git@github.com:company/repo-a.git
    dependencies:
      - agent: document_processor
        version: ">=1.2.0"
    environments:
      dev: { branch: dev, version: v2.1.0, status: configured }

How This Illustrative Approach Works

The pipeline reads agent-registry.yaml
It builds a dependency graph such as document_processor → employee_onboarding
It deploys in order: first document_processor, then employee_onboarding
It validates version requirements before each deployment
It updates the registry with new versions and timestamps

Why This Simple Approach Can Work

Single Active Version

One active version per environment simplifies tracking

No Extra Infrastructure

No server infrastructure is required beyond a YAML file in Git

Version Controlled

Git tracks every registry change

Dependency Validation

Helps prevent incompatible deployments

Alternative Approaches

Option 1: More Sophisticated Solutions

Dedicated dependency management service
Integration with enterprise service catalog
Advanced version resolution algorithms
Real-time dependency validation APIs

Option 2: Simpler Solutions

Manual coordination via documentation
Orchestrated multi-repository pipeline without a registry
Git submodules for dependent agent definitions

Option 3: No Explicit Dependency Management

Appropriate when agents are loosely coupled and failures are acceptable
Rely on runtime error handling and monitoring

8) Testing and Quality Gates

Unit tests: knowledge bases, tools, and flows
Static validation: JSON Schema for agent.yaml and *-releases.yaml
Contract tests: tool I/O and validation that agents reference valid tool versions
Prompt regression: low-temperature runs to validate tool selection and output format
Security: SAST and dependency scans
Policy: OPA or Conftest rules, for example forbidding latest, enforcing approved model IDs, and applying temperature caps in production

8) Git-Based Promotion and Rollback

Promotion Workflow via Branch Merging

Promotions follow a strict path: dev → qa → preprod → prod All promotions are performed through Git branch merges by using pull requests.

Within-Tenant Promotions

Dev to QA
PreProd to Prod

Dev → QA within dev-qa-tenant

# 1. Develop and test in dev branch (dev-qa-tenant Draft)
# 2. Create PR: dev → qa
# 3. PR review + approval
# 4. Merge triggers Jenkins pipeline
# 5. Auto-deploys to dev-qa-tenant Live (QA)
# 6. Git tag created: v1.2.3-qa

# Example PR
git checkout qa
git pull origin qa
git merge --no-ff dev
git push origin qa

PreProd → Prod within prod-tenant

# 1. Validate in preprod branch (prod-tenant Draft)
# 2. Create PR: preprod → prod
# 3. Requires CAB approval + change ticket
# 4. Merge triggers Jenkins pipeline
# 5. Deploys to prod-tenant Live (Prod)
# 6. Git tag created: v1.2.3-prod

# Example PR with approval
git checkout prod
git pull origin prod
git merge --no-ff preprod
git push origin prod

Cross-Tenant Promotion

QA → PreProd crosses the tenant boundary from dev-qa-tenant to prod-tenant.

# This is the critical cross-tenant boundary
# 1. Validate in qa branch (dev-qa-tenant Live)
# 2. Create PR: qa → preprod
# 3. Requires team lead approval
# 4. Merge triggers Jenkins pipeline
# 5. Pipeline switches authentication from dev-qa-tenant to prod-tenant
# 6. Deploys to prod-tenant Draft (PreProd)
# 7. Git tag created: v1.2.3-preprod

# Example cross-tenant PR
git checkout preprod
git pull origin preprod
git merge --no-ff qa
# Resolve any conflicts (e.g., config/tenant.yaml, config/values.yaml)
git push origin preprod

Handling Merge Conflicts

# Common conflicts during cross-tenant merge (qa → preprod):
# - config/tenant.yaml (keep preprod's tenant config)
# - config/values.yaml (keep preprod's environment values)
# - connections/ (may differ between tenants)

# Resolution strategy:
git checkout preprod
git merge qa
# CONFLICT in config/tenant.yaml
git checkout --theirs config/tenant.yaml  # Keep preprod's tenant config
git checkout --theirs config/values.yaml  # Keep preprod's values
git add config/
git commit -m "Merge qa → preprod: Promote v1.2.3"

Promotion Best Practices

Always use pull requests: never push directly to environment branches
Use merge commits: use --no-ff to preserve promotion history
Enforce approval gates
- dev → qa: automated after CI checks pass
- qa → preprod: team lead approval for the cross-tenant boundary
- preprod → prod: CAB approval and a change ticket
Run smoke tests after each deployment before marking success
Tag only after success: Git tags are created automatically by Jenkins after successful deployment
Review diffs: use git diff qa..preprod to inspect what will be promoted

Comparing Branches Before Promotion

# See what changes will be promoted from qa to preprod
git diff qa..preprod

# See commit history
git log preprod..qa --oneline

# See file-level changes
git diff qa..preprod --stat

# See specific file changes
git diff qa..preprod -- agents/employee_onboarding/agent.yaml

Rollback Strategies

Knowledge-Base Promotion and Rollback

Knowledge bases require special handling during promotion and rollback. Promotion workflow:

Knowledge bases are not promoted from Draft to Live because that concept does not exist
During cross-tenant promotion (qa → staging), knowledge bases must be redeployed to prod-tenant
Git merges bring the knowledge-base YAML files forward, but CI/CD must still execute deployment

Rollback considerations:

Rolling back an agent does not automatically roll back its knowledge base
Rolling back a knowledge base requires explicit redeployment of the previous version

# Checkout previous knowledge-base version
git checkout v1.2.0 -- knowledge-bases/hr_policy_docs.yaml

# Redeploy to tenant
orchestrate knowledge-bases import -f knowledge-bases/hr_policy_docs.yaml

A knowledge-base rollback affects both Draft and Live agents in the same tenant immediately.

Simple Rollback (Revert Merge)

# Rollback prod to previous state
git checkout prod
git revert -m 1 <merge-commit-sha>  # Revert the merge commit
git push origin prod
# Triggers pipeline to redeploy previous version

Example:

# Find the merge commit to revert
git log prod --oneline --merges
# Output: abc1234 Merge preprod → prod: v1.2.3

# Revert the merge
git revert -m 1 abc1234
git push origin prod
# Jenkins redeploys the previous version (v1.2.2-prod)

Rollback to Specific Tag

# Rollback prod to a specific known-good version
git checkout prod
git reset --hard v1.2.0-prod  # Reset to specific tag
git push --force origin prod  # Force push (requires special permissions)
# Triggers pipeline to redeploy v1.2.0

Force-pushing requires special permissions and should be used only in emergency situations.

Emergency Rollback (Cross-Tenant)

If Production (prod-tenant Live) has critical issues:

# 1. Identify last known good tag
git tag -l "v*-prod" --sort=-version:refname | head -5

# 2. Create emergency rollback branch
git checkout -b emergency-rollback-prod v1.2.0-prod

# 3. Force restore to prod (with approvals)
git checkout prod
git reset --hard emergency-rollback-prod
git push --force origin prod

# 4. Pipeline redeploys v1.2.0 to prod-tenant Live
# 5. Post-mortem to understand root cause

Rollback Considerations

Git history: all rollbacks remain visible in Git history
Backward compatibility: verify that old versions still work with current infrastructure
Database migrations: might require manual intervention when schema changes exist
Connection changes: verify that old versions still work with current connection settings
Tenant isolation: a rollback in prod-tenant does not affect dev-qa-tenant
Tag references: use tags to identify exact versions for rollback

Promotion Checklist

Before promoting `qa → preprod`

Before promoting `preprod → prod`

Git Workflow Diagram

main (development)
  ↓ (feature branches merge here)
dev branch
  ↓ (PR: dev → qa)
qa branch
  ↓ (PR: qa → preprod) ← CROSS-TENANT BOUNDARY
preprod branch
  ↓ (PR: preprod → prod)
prod branch

Tags:
v1.2.3-dev → v1.2.3-qa → v1.2.3-preprod → v1.2.3-prod

9) Security and Governance

Secrets: use secret managers only; never commit secrets; rotate regularly
RBAC: restrict Jenkins credentials and use scoped tokens for the wxO API
Network: use allowlists for API endpoints
Images: if using image mode, sign images and avoid :latest
Reviews: use code owners for agents/ and envs/prod/

11) wxO ADK CLI Deployment Commands

This section provides examples of using the watsonx Orchestrate ADK CLI to deploy agents, flows, and tools to different environments and tenants.

Prerequisites

Install wxO ADK: pip install ibm-watsonx-orchestrate
Activate environment: point the CLI to the correct tenant
Authenticate: ensure the proper credentials are configured

Environment Activation

Before deploying, activate the correct tenant environment:

# Activate dev-qa-tenant for Dev/QA deployments
orchestrate env activate dev-qa-tenant

# Activate prod-tenant for PreProd/Prod deployments
orchestrate env activate prod-tenant

Deploying Tools

Tools must be deployed before agents that use them.

Python tool
Flow
OpenAPI tool

# Deploy a Python tool
orchestrate tools import \
  -k python \
  -f tools/hr_api/tool.py

# Deploy an agentic workflow tool
orchestrate tools import \
  -k flow \
  -f tools/document_processing/workflow.py

# Deploy an OpenAPI tool
orchestrate tools import \
  -k openapi \
  -f tools/db_query/openapi.yaml

Deploying Knowledge Bases

Knowledge bases must be deployed before an agent can use them.

Built-in knowledge base
External knowledge base

orchestrate knowledge-bases import -f knowledge-bases/hr_policy_docs.yaml

orchestrate knowledge-bases import -f knowledge-bases/product_catalog.yaml -a elastic_search_creds

Agent Deployment Commands

Agents use two different commands depending on the target environment.

1. `orchestrate agents import`

Used for Dev and PreProd Draft environments
Imports the hand-written agent YAML into Draft

orchestrate agents import -f agents/employee_onboarding/agent.yaml

2. `orchestrate agents deploy`

Used for QA and Prod Live environments
Promotes an existing agent from Draft to Live within the same tenant
See Deploying an agent

orchestrate agents deploy \
  --name employee_onboarding

Deployment Flow

Deploy flows first if the agent uses them
For Dev and PreProd, use import to deploy the agent to Draft
- Tools are automatically deployed based on the agent’s tools: section
Test the agent in Draft
For QA and Prod, use deploy to promote from Draft to Live

Agent YAML Creation

Agent YAML files are hand-written and stored in Git
Do not use orchestrate agents create to generate YAML
YAML files serve as the source of truth for agent configuration
This supports version control and code review for agent definitions

Deployment Examples by Environment

Dev
QA
PreProd
Prod

Dev Deployment (dev-qa-tenant, Draft)

# 1. Activate dev-qa-tenant
orchestrate env activate dev-qa-tenant

# 2. Deploy flows (if any)
orchestrate tools import -k flow -f tools/document_processing/workflow.py

# 3. Deploy knowledge-bases (if any)
orchestrate knowledge-bases import -f knowledge-bases/hr_policy_docs.yaml

# 4. Deploy agent (tools auto-deploy via agent's tools: section)
orchestrate agents import -f agents/employee_onboarding/agent.yaml
# - Agent will be in Draft state in dev-qa-tenant

QA Deployment (dev-qa-tenant, Live)

# 1. Ensure dev-qa-tenant is active
orchestrate env activate dev-qa-tenant

# 2. Deploy updated flows (if changed)
orchestrate tools import -k flow -f tools/document_processing/workflow.py

# 3. Deploy updated knowledge-bases (if changed)
orchestrate knowledge-bases import -f knowledge-bases/hr_policy_docs.yaml
# Note: knowledge-bases do not differentiate between draft and live environments.
# A knowledge-base update applies to both Draft and Live simultaneously.

# 4. Deploy agent to Live (QA) - promotes from Draft to Live
orchestrate agents deploy \
  --name employee_onboarding

PreProd Deployment (prod-tenant, Draft)

# 1. Switch to prod-tenant (CROSS-TENANT BOUNDARY)
orchestrate env activate prod-tenant

# 2. Deploy flows to prod-tenant (if any)
orchestrate tools import -k flow -f tools/document_processing/workflow.py

# 3. Deploy knowledge-bases (if any)
orchestrate knowledge-bases import -f knowledge-bases/hr_policy_docs.yaml

# 4. Deploy agent to prod-tenant (tools auto-deploy via agent's tools: section)
orchestrate agents import -f agents/employee_onboarding/agent.yaml
# - Agent will be in Draft state in prod-tenant (PreProd)

Prod Deployment (prod-tenant, Live)

# 1. Ensure prod-tenant is active
orchestrate env activate prod-tenant

# 2. Verify flows and agent are deployed

# 3. Verify agent is tested in Draft (PreProd)

# 4. Deploy agent to Live (Prod) - promotes from Draft to Live
orchestrate agents deploy \
  --name employee_onboarding

# Note: Requires additional approvals (CAB, change ticket)
# Note: knowledge-bases do not differentiate between draft and live environments.
# A knowledge-base update applies to both Draft and Live simultaneously.

Complete Deployment Script Example

#!/bin/bash
set -e

# Deployment script for a specific environment
ENVIRONMENT=$1  # dev, qa, preprod, or prod
BRANCH=$(git branch --show-current)

echo "Deploying branch: $BRANCH to environment: $ENVIRONMENT"

# Determine tenant based on environment
if [ "$ENVIRONMENT" = "dev" ] || [ "$ENVIRONMENT" = "qa" ]; then
  TENANT="dev-qa-tenant"
elif [ "$ENVIRONMENT" = "preprod" ] || [ "$ENVIRONMENT" = "prod" ]; then
  TENANT="prod-tenant"
else
  echo "Invalid environment: $ENVIRONMENT"
  exit 1
fi

echo "Activating tenant: $TENANT"
orchestrate env activate $TENANT

# Deploy all knowledge-bases (affects both Draft and Live)
echo "Deploying knowledge-bases (affects both Draft and Live)..."
for kb_file in knowledge-bases/*.yaml; do
  if [ -f "$kb_file" ]; then
    kb_name=$(basename "$kb_file")
    echo "Deploying knowledge-base: $kb_name"
    orchestrate knowledge-bases import -f "$kb_file"
  fi
done

# Deploy all flows from tools/ directory (other tools auto-deploy via agents)
echo "Deploying flows..."
for tool_dir in tools/*/; do
  tool_name=$(basename "$tool_dir")
  if [ -f "$tool_dir/workflow.py" ]; then
    echo "Deploying flow: $tool_name"
    orchestrate tools import -k flow -f "$tool_dir/workflow.py"
  fi
done

# Deploy all agents (tools auto-deploy based on agent's tools: section)
echo "Deploying agents..."
for agent_dir in agents/*/; do
  agent_name=$(basename "$agent_dir")
  if [ -f "$agent_dir/agent.yaml" ]; then
    echo "Deploying agent: $agent_name"
    if [ "$ENVIRONMENT" = "dev" ] || [ "$ENVIRONMENT" = "preprod" ]; then
      # Deploy to Draft environment
      orchestrate agents import -f "$agent_dir/agent.yaml"
    elif [ "$ENVIRONMENT" = "qa" ] || [ "$ENVIRONMENT" = "prod" ]; then
      # Promote from Draft to Live
      orchestrate agents deploy --name "$agent_name"
    fi
  fi
done

echo "Note: Tools are automatically deployed when agents reference them in tools: section"

echo "Deployment to $ENVIRONMENT ($TENANT) completed successfully!"

Usage

# Deploy develop branch to dev environment
./deploy.sh dev

# Deploy qa-1.2.0 branch to qa environment
./deploy.sh qa

# Deploy staging-1.2.0 branch to staging environment
./deploy.sh staging

# Deploy main branch to prod environment
./deploy.sh prod

Key Differences by Tenant

Aspect	dev-qa-tenant	prod-tenant
Activation	`orchestrate env activate dev-qa-tenant`	`orchestrate env activate prod-tenant`
Environments	Draft (Dev), Live (QA)	Draft (PreProd), Live (Prod)
Credentials	Dev/QA API keys	Prod API keys
Connections	`connections/dev-qa-tenant/`	`connections/prod-tenant/`
Approval	Automated	Requires CAB approval

Additional CLI Commands

List agents
Export agent
Delete agent
List tools
Export tool

orchestrate agents list

orchestrate agents export -n employee_onboarding -k native -o agent-backup.yaml

orchestrate agents delete -n employee_onboarding

orchestrate tools list

orchestrate tools export -n hr_api -o hr-api-backup.zip

For complete CLI documentation, see the watsonx Orchestrate developer documentation.

11) Summary

Core Principles

Apply GitOps principles by using Git branches for versioning instead of manual version directories
Replace Argo CD with a Jenkins multi-branch pipeline as the orchestrator
Use the wxO CLI to deploy agents, flows, tools, and knowledge bases through orchestrate commands
Keep agent YAML files hand-written and stored in Git
Treat knowledge bases as tenant-scoped and not environment-scoped
Maintain secret-free Git through connections.template.yaml plus secret substitution at deploy time
Achieve traceable promotions through Git merges, safe rollbacks through Git reverts, and consistent deployments across environments without direct cluster access

Git-Based Versioning

Git branches represent environment state such as dev, qa, preprod, and prod
Git tags mark deployments such as v1.2.3-dev and v1.2.3-qa
Git merges promote code between environments
No manual version management is required because Git handles versioning
Natural evolution is possible because agent structures can change between environments

Two-Tenant Architecture

Tenant 1
Tenant 2

dev-qa-tenant for non-production workloads

Draft → Dev (develop)
Live → QA (qa-<version>)

prod-tenant for production workloads

Draft → Staging (staging-<version>)
Live → Production (main)

Key Benefits

Complete Isolation

Production infrastructure remains separated from development and testing

Security

Different credentials, RBAC, and network policies per tenant

Compliance

Separate audit trails for production and non-production

Risk Mitigation

Issues in Dev or QA cannot impact PreProd or Prod

Flexibility

Each tenant can have different configurations, quotas, and policies

Simplified Versioning

Git branches and tags replace manual version directories

Natural Evolution

Code structure can evolve through branches

Easy Comparison

git diff shows exactly what will be promoted

Promotion Path

develop → dev-qa-tenant Draft
  ↓ (PR: develop → qa-<version>)
qa-<version> → dev-qa-tenant Live
  ↓ (PR: qa-<version> → staging-<version>) ← CROSS-TENANT BOUNDARY
staging-<version> → prod-tenant Draft
  ↓ (PR: staging-<version> → main)
main → prod-tenant Live (PRODUCTION)

Git Tags:
v1.2.3-dev → v1.2.3-qa → v1.2.3-staging → v1.2.3-prod

Critical Cross-Tenant Transition

The qa-<version> → staging-<version> merge is the critical boundary where:

The pipeline switches from dev-qa-tenant authentication to prod-tenant
API endpoints and credentials change
Configuration files such as tenant.yaml and values.yaml may require conflict resolution
Additional approval gates apply
Production-grade validation begins

Release Notes

Get Started

Build

Deploy

Analyze

Developer experience

Legal notices

​7) Inter-Agent Dependencies (Multi-Repository)

​Overview

​Core Principle: CI/CD Pipeline as the Only Authorized Deployment Path

​Key Challenges

​Illustrative Solution: File-Based Agent Registry

​Example Registry Structure

​How This Illustrative Approach Works

​Why This Simple Approach Can Work

Single Active Version

No Extra Infrastructure

Version Controlled

Dependency Validation

​Alternative Approaches

​8) Testing and Quality Gates

​8) Git-Based Promotion and Rollback

​Promotion Workflow via Branch Merging

​Within-Tenant Promotions

​Cross-Tenant Promotion

​Handling Merge Conflicts

​Promotion Best Practices

​Comparing Branches Before Promotion

​Rollback Strategies

​Knowledge-Base Promotion and Rollback

​Simple Rollback (Revert Merge)

​Rollback to Specific Tag

​Emergency Rollback (Cross-Tenant)

​Rollback Considerations

​Promotion Checklist

​Before promoting qa → preprod

​Before promoting preprod → prod

​Git Workflow Diagram

​9) Security and Governance

​11) wxO ADK CLI Deployment Commands

​Prerequisites

​Environment Activation

​Deploying Tools

​Deploying Knowledge Bases

​Agent Deployment Commands

​1. orchestrate agents import

​2. orchestrate agents deploy

​Deployment Flow

​Agent YAML Creation

​Deployment Examples by Environment

​Complete Deployment Script Example

​Usage

​Key Differences by Tenant

​Additional CLI Commands

​11) Summary

​Core Principles

​Git-Based Versioning

​Two-Tenant Architecture

​Key Benefits

Complete Isolation

Security

Compliance

Risk Mitigation

Flexibility

Simplified Versioning

Natural Evolution

Easy Comparison

​Promotion Path

​Critical Cross-Tenant Transition

​Implementation Checklist

7) Inter-Agent Dependencies (Multi-Repository)

Overview

Core Principle: CI/CD Pipeline as the Only Authorized Deployment Path

Key Challenges

Illustrative Solution: File-Based Agent Registry

Example Registry Structure

How This Illustrative Approach Works

Why This Simple Approach Can Work

Alternative Approaches

8) Testing and Quality Gates

8) Git-Based Promotion and Rollback

Promotion Workflow via Branch Merging

Within-Tenant Promotions

Cross-Tenant Promotion

Handling Merge Conflicts

Promotion Best Practices

Comparing Branches Before Promotion

Rollback Strategies

Knowledge-Base Promotion and Rollback

Simple Rollback (Revert Merge)

Rollback to Specific Tag

Emergency Rollback (Cross-Tenant)

Rollback Considerations

Promotion Checklist

Before promoting `qa → preprod`

Before promoting `preprod → prod`

Git Workflow Diagram

9) Security and Governance

11) wxO ADK CLI Deployment Commands

Prerequisites

Environment Activation

Deploying Tools

Deploying Knowledge Bases

Agent Deployment Commands

1. `orchestrate agents import`

2. `orchestrate agents deploy`

Deployment Flow

Agent YAML Creation

Deployment Examples by Environment

Complete Deployment Script Example

Usage

Key Differences by Tenant

Additional CLI Commands

11) Summary

Core Principles

Git-Based Versioning

Two-Tenant Architecture

Key Benefits

Promotion Path

Critical Cross-Tenant Transition

Implementation Checklist