> ## Documentation Index
> Fetch the complete documentation index at: https://developer.watson-orchestrate.ibm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# wxO CI/CD Deployment Approach (Part 3)

> Inter-agent dependency patterns, testing gates, promotion and rollback workflows, security controls, and CLI deployment examples for wxO CI/CD

## 7) Inter-Agent Dependencies (Multi-Repository)

### Overview

When agents are managed in **separate Git repositories** and have dependencies on each other, such as AgentA depending on AgentB, you need a strategy to coordinate deployments across repositories.

### Core Principle: CI/CD Pipeline as the Only Authorized Deployment Path

The general approach is to use the **CI/CD pipeline as the only authorized path** to deploy agents to production environments. Customers should **not deploy agents manually** by hand or with ad hoc scripts. This ensures:

* **Consistency**: all deployments follow the same process
* **Auditability**: every deployment is tracked in CI/CD logs
* **Governance**: approval gates and quality checks are enforced
* **Traceability**: deployments are linked to Git commits and pull requests

### Key Challenges

* **Version synchronization**: dependent agents must reference compatible versions
* **Deployment order**: dependencies must be deployed before dependents
* **Cross-tenant coordination**: agents must exist in the same tenant and environment
* **Testing**: integration tests require all dependent agents to be available
* **Single active version**: only one version of an agent or tool is active in a given environment

### Illustrative Solution: File-Based Agent Registry

<Warning>
  The file-based registry approach described below is **one possible solution for illustration purposes only**. It is **not a recommendation or best practice**. The final solution might be more sophisticated or simpler depending on the level of lifecycle governance required by your organization.
</Warning>

Given that only **one version of an agent or tool is active** in a given environment, a simple **file-based agent registry** can be used to maintain dependencies and validate deployments.

#### Example Registry Structure

```yaml theme={null}
# agent-registry.yaml - File-based catalog for multi-repo dependencies
# This is an ILLUSTRATIVE example, not a prescriptive solution
agents:
  document_processor:
    repository: git@github.com:company/repo-b.git
    dependencies: []  # No dependencies - deploys first
    environments:
      dev: { branch: dev, version: v1.2.3, status: configured }
  
  employee_onboarding:
    repository: git@github.com:company/repo-a.git
    dependencies:
      - agent: document_processor
        version: ">=1.2.0"
    environments:
      dev: { branch: dev, version: v2.1.0, status: configured }
```

#### How This Illustrative Approach Works

1. The pipeline reads `agent-registry.yaml`
2. It builds a dependency graph such as `document_processor` → `employee_onboarding`
3. It deploys in order: first `document_processor`, then `employee_onboarding`
4. It validates version requirements before each deployment
5. It updates the registry with new versions and timestamps

### Why This Simple Approach Can Work

<CardGroup cols={2}>
  <Card title="Single Active Version" icon="code-branch">
    One active version per environment simplifies tracking
  </Card>

  <Card title="No Extra Infrastructure" icon="server">
    No server infrastructure is required beyond a YAML file in Git
  </Card>

  <Card title="Version Controlled" icon="git-alt">
    Git tracks every registry change
  </Card>

  <Card title="Dependency Validation" icon="shield-check">
    Helps prevent incompatible deployments
  </Card>
</CardGroup>

### Alternative Approaches

<AccordionGroup>
  <Accordion title="Option 1: More Sophisticated Solutions">
    * Dedicated dependency management service
    * Integration with enterprise service catalog
    * Advanced version resolution algorithms
    * Real-time dependency validation APIs
  </Accordion>

  <Accordion title="Option 2: Simpler Solutions">
    * Manual coordination via documentation
    * Orchestrated multi-repository pipeline without a registry
    * Git submodules for dependent agent definitions
  </Accordion>

  <Accordion title="Option 3: No Explicit Dependency Management">
    * Appropriate when agents are loosely coupled and failures are acceptable
    * Rely on runtime error handling and monitoring
  </Accordion>
</AccordionGroup>

## 8) Testing and Quality Gates

* **Unit tests**: knowledge bases, tools, and flows
* **Static validation**: JSON Schema for `agent.yaml` and `*-releases.yaml`
* **Contract tests**: tool I/O and validation that agents reference valid tool versions
* **Prompt regression**: low-temperature runs to validate tool selection and output format
* **Security**: SAST and dependency scans
* **Policy**: OPA or Conftest rules, for example forbidding `latest`, enforcing approved model IDs, and applying temperature caps in production

## 8) Git-Based Promotion and Rollback

### Promotion Workflow via Branch Merging

Promotions follow a strict path: **dev → qa → preprod → prod**

All promotions are performed through **Git branch merges** by using pull requests.

### Within-Tenant Promotions

<Tabs>
  <Tab title="Dev to QA">
    **Dev → QA** within `dev-qa-tenant`

    ```bash theme={null}
    # 1. Develop and test in dev branch (dev-qa-tenant Draft)
    # 2. Create PR: dev → qa
    # 3. PR review + approval
    # 4. Merge triggers Jenkins pipeline
    # 5. Auto-deploys to dev-qa-tenant Live (QA)
    # 6. Git tag created: v1.2.3-qa

    # Example PR
    git checkout qa
    git pull origin qa
    git merge --no-ff dev
    git push origin qa
    ```
  </Tab>

  <Tab title="PreProd to Prod">
    **PreProd → Prod** within `prod-tenant`

    ```bash theme={null}
    # 1. Validate in preprod branch (prod-tenant Draft)
    # 2. Create PR: preprod → prod
    # 3. Requires CAB approval + change ticket
    # 4. Merge triggers Jenkins pipeline
    # 5. Deploys to prod-tenant Live (Prod)
    # 6. Git tag created: v1.2.3-prod

    # Example PR with approval
    git checkout prod
    git pull origin prod
    git merge --no-ff preprod
    git push origin prod
    ```
  </Tab>
</Tabs>

### Cross-Tenant Promotion

**QA → PreProd** crosses the tenant boundary from `dev-qa-tenant` to `prod-tenant`.

```bash theme={null}
# This is the critical cross-tenant boundary
# 1. Validate in qa branch (dev-qa-tenant Live)
# 2. Create PR: qa → preprod
# 3. Requires team lead approval
# 4. Merge triggers Jenkins pipeline
# 5. Pipeline switches authentication from dev-qa-tenant to prod-tenant
# 6. Deploys to prod-tenant Draft (PreProd)
# 7. Git tag created: v1.2.3-preprod

# Example cross-tenant PR
git checkout preprod
git pull origin preprod
git merge --no-ff qa
# Resolve any conflicts (e.g., config/tenant.yaml, config/values.yaml)
git push origin preprod
```

### Handling Merge Conflicts

```bash theme={null}
# Common conflicts during cross-tenant merge (qa → preprod):
# - config/tenant.yaml (keep preprod's tenant config)
# - config/values.yaml (keep preprod's environment values)
# - connections/ (may differ between tenants)

# Resolution strategy:
git checkout preprod
git merge qa
# CONFLICT in config/tenant.yaml
git checkout --theirs config/tenant.yaml  # Keep preprod's tenant config
git checkout --theirs config/values.yaml  # Keep preprod's values
git add config/
git commit -m "Merge qa → preprod: Promote v1.2.3"
```

### Promotion Best Practices

1. **Always use pull requests**: never push directly to environment branches
2. **Use merge commits**: use `--no-ff` to preserve promotion history
3. **Enforce approval gates**
   * `dev → qa`: automated after CI checks pass
   * `qa → preprod`: team lead approval for the cross-tenant boundary
   * `preprod → prod`: CAB approval and a change ticket
4. **Run smoke tests** after each deployment before marking success
5. **Tag only after success**: Git tags are created automatically by Jenkins after successful deployment
6. **Review diffs**: use `git diff qa..preprod` to inspect what will be promoted

### Comparing Branches Before Promotion

```bash theme={null}
# See what changes will be promoted from qa to preprod
git diff qa..preprod

# See commit history
git log preprod..qa --oneline

# See file-level changes
git diff qa..preprod --stat

# See specific file changes
git diff qa..preprod -- agents/employee_onboarding/agent.yaml
```

### Rollback Strategies

#### Knowledge-Base Promotion and Rollback

Knowledge bases require special handling during promotion and rollback.

**Promotion workflow:**

* Knowledge bases are **not promoted** from Draft to Live because that concept does not exist
* During cross-tenant promotion (`qa → staging`), knowledge bases must be **redeployed** to `prod-tenant`
* Git merges bring the knowledge-base YAML files forward, but CI/CD must still execute deployment

**Rollback considerations:**

* Rolling back an agent does **not** automatically roll back its knowledge base
* Rolling back a knowledge base requires explicit redeployment of the previous version

```bash theme={null}
# Checkout previous knowledge-base version
git checkout v1.2.0 -- knowledge-bases/hr_policy_docs.yaml

# Redeploy to tenant
orchestrate knowledge-bases import -f knowledge-bases/hr_policy_docs.yaml
```

<Warning>
  A knowledge-base rollback affects **both Draft and Live** agents in the same tenant immediately.
</Warning>

#### Simple Rollback (Revert Merge)

```bash theme={null}
# Rollback prod to previous state
git checkout prod
git revert -m 1 <merge-commit-sha>  # Revert the merge commit
git push origin prod
# Triggers pipeline to redeploy previous version
```

**Example:**

```bash theme={null}
# Find the merge commit to revert
git log prod --oneline --merges
# Output: abc1234 Merge preprod → prod: v1.2.3

# Revert the merge
git revert -m 1 abc1234
git push origin prod
# Jenkins redeploys the previous version (v1.2.2-prod)
```

#### Rollback to Specific Tag

```bash theme={null}
# Rollback prod to a specific known-good version
git checkout prod
git reset --hard v1.2.0-prod  # Reset to specific tag
git push --force origin prod  # Force push (requires special permissions)
# Triggers pipeline to redeploy v1.2.0
```

<Warning>
  Force-pushing requires special permissions and should be used only in emergency situations.
</Warning>

#### Emergency Rollback (Cross-Tenant)

If Production (`prod-tenant` Live) has critical issues:

```bash theme={null}
# 1. Identify last known good tag
git tag -l "v*-prod" --sort=-version:refname | head -5

# 2. Create emergency rollback branch
git checkout -b emergency-rollback-prod v1.2.0-prod

# 3. Force restore to prod (with approvals)
git checkout prod
git reset --hard emergency-rollback-prod
git push --force origin prod

# 4. Pipeline redeploys v1.2.0 to prod-tenant Live
# 5. Post-mortem to understand root cause
```

### Rollback Considerations

* **Git history**: all rollbacks remain visible in Git history
* **Backward compatibility**: verify that old versions still work with current infrastructure
* **Database migrations**: might require manual intervention when schema changes exist
* **Connection changes**: verify that old versions still work with current connection settings
* **Tenant isolation**: a rollback in `prod-tenant` does not affect `dev-qa-tenant`
* **Tag references**: use tags to identify exact versions for rollback

### Promotion Checklist

#### Before promoting `qa → preprod`

* [ ] All QA tests passed
* [ ] Security scan completed
* [ ] Performance benchmarks met
* [ ] `git diff qa..preprod` reviewed
* [ ] Merge conflicts resolved, especially configuration files
* [ ] Documentation updated
* [ ] Runbooks reviewed
* [ ] Rollback plan documented
* [ ] Change ticket approved
* [ ] Stakeholders notified

#### Before promoting `preprod → prod`

* [ ] PreProd validation completed
* [ ] No critical issues in PreProd
* [ ] Load testing completed
* [ ] `git diff preprod..prod` reviewed
* [ ] Disaster recovery tested
* [ ] On-call team briefed
* [ ] Maintenance window scheduled if needed
* [ ] CAB approval obtained
* [ ] Emergency rollback plan ready

### Git Workflow Diagram

```text theme={null}
main (development)
  ↓ (feature branches merge here)
dev branch
  ↓ (PR: dev → qa)
qa branch
  ↓ (PR: qa → preprod) ← CROSS-TENANT BOUNDARY
preprod branch
  ↓ (PR: preprod → prod)
prod branch

Tags:
v1.2.3-dev → v1.2.3-qa → v1.2.3-preprod → v1.2.3-prod
```

## 9) Security and Governance

* **Secrets**: use secret managers only; never commit secrets; rotate regularly
* **RBAC**: restrict Jenkins credentials and use scoped tokens for the wxO API
* **Network**: use allowlists for API endpoints
* **Images**: if using image mode, sign images and avoid `:latest`
* **Reviews**: use code owners for `agents/` and `envs/prod/`

## 11) wxO ADK CLI Deployment Commands

This section provides examples of using the **watsonx Orchestrate ADK CLI** to deploy agents, flows, and tools to different environments and tenants.

### Prerequisites

1. **Install wxO ADK**: `pip install ibm-watsonx-orchestrate`
2. **Activate environment**: point the CLI to the correct tenant
3. **Authenticate**: ensure the proper credentials are configured

### Environment Activation

Before deploying, activate the correct tenant environment:

```bash theme={null}
# Activate dev-qa-tenant for Dev/QA deployments
orchestrate env activate dev-qa-tenant

# Activate prod-tenant for PreProd/Prod deployments
orchestrate env activate prod-tenant
```

### Deploying Tools

Tools must be deployed before agents that use them.

<Tabs>
  <Tab title="Python tool">
    ```bash theme={null}
    # Deploy a Python tool
    orchestrate tools import \
      -k python \
      -f tools/hr_api/tool.py
    ```
  </Tab>

  <Tab title="Flow">
    ```bash theme={null}
    # Deploy an agentic workflow tool
    orchestrate tools import \
      -k flow \
      -f tools/document_processing/workflow.py
    ```
  </Tab>

  <Tab title="OpenAPI tool">
    ```bash theme={null}
    # Deploy an OpenAPI tool
    orchestrate tools import \
      -k openapi \
      -f tools/db_query/openapi.yaml
    ```
  </Tab>
</Tabs>

### Deploying Knowledge Bases

Knowledge bases must be deployed before an agent can use them.

<Tabs>
  <Tab title="Built-in knowledge base">
    ```bash theme={null}
    orchestrate knowledge-bases import -f knowledge-bases/hr_policy_docs.yaml
    ```
  </Tab>

  <Tab title="External knowledge base">
    ```bash theme={null}
    orchestrate knowledge-bases import -f knowledge-bases/product_catalog.yaml -a elastic_search_creds
    ```
  </Tab>
</Tabs>

### Agent Deployment Commands

<Info>
  Agents use **two different commands** depending on the target environment.
</Info>

#### 1. `orchestrate agents import`

* Used for **Dev** and **PreProd** Draft environments
* Imports the hand-written agent YAML into Draft

```bash theme={null}
orchestrate agents import -f agents/employee_onboarding/agent.yaml
```

#### 2. `orchestrate agents deploy`

* Used for **QA** and **Prod** Live environments
* Promotes an existing agent from Draft to Live within the same tenant
* See [Deploying an agent](/agents/deploy_agent)

```bash theme={null}
orchestrate agents deploy \
  --name employee_onboarding
```

### Deployment Flow

1. Deploy flows first if the agent uses them
2. For **Dev** and **PreProd**, use `import` to deploy the agent to Draft
   * Tools are automatically deployed based on the agent's `tools:` section
3. Test the agent in Draft
4. For **QA** and **Prod**, use `deploy` to promote from Draft to Live

### Agent YAML Creation

* Agent YAML files are **hand-written** and stored in Git
* Do **not** use `orchestrate agents create` to generate YAML
* YAML files serve as the **source of truth** for agent configuration
* This supports version control and code review for agent definitions

### Deployment Examples by Environment

<Tabs>
  <Tab title="Dev">
    **Dev Deployment** (`dev-qa-tenant`, Draft)

    ```bash theme={null}
    # 1. Activate dev-qa-tenant
    orchestrate env activate dev-qa-tenant

    # 2. Deploy flows (if any)
    orchestrate tools import -k flow -f tools/document_processing/workflow.py

    # 3. Deploy knowledge-bases (if any)
    orchestrate knowledge-bases import -f knowledge-bases/hr_policy_docs.yaml

    # 4. Deploy agent (tools auto-deploy via agent's tools: section)
    orchestrate agents import -f agents/employee_onboarding/agent.yaml
    # - Agent will be in Draft state in dev-qa-tenant
    ```
  </Tab>

  <Tab title="QA">
    **QA Deployment** (`dev-qa-tenant`, Live)

    ```bash theme={null}
    # 1. Ensure dev-qa-tenant is active
    orchestrate env activate dev-qa-tenant

    # 2. Deploy updated flows (if changed)
    orchestrate tools import -k flow -f tools/document_processing/workflow.py

    # 3. Deploy updated knowledge-bases (if changed)
    orchestrate knowledge-bases import -f knowledge-bases/hr_policy_docs.yaml
    # Note: knowledge-bases do not differentiate between draft and live environments.
    # A knowledge-base update applies to both Draft and Live simultaneously.

    # 4. Deploy agent to Live (QA) - promotes from Draft to Live
    orchestrate agents deploy \
      --name employee_onboarding
    ```
  </Tab>

  <Tab title="PreProd">
    **PreProd Deployment** (`prod-tenant`, Draft)

    ```bash theme={null}
    # 1. Switch to prod-tenant (CROSS-TENANT BOUNDARY)
    orchestrate env activate prod-tenant

    # 2. Deploy flows to prod-tenant (if any)
    orchestrate tools import -k flow -f tools/document_processing/workflow.py

    # 3. Deploy knowledge-bases (if any)
    orchestrate knowledge-bases import -f knowledge-bases/hr_policy_docs.yaml

    # 4. Deploy agent to prod-tenant (tools auto-deploy via agent's tools: section)
    orchestrate agents import -f agents/employee_onboarding/agent.yaml
    # - Agent will be in Draft state in prod-tenant (PreProd)
    ```
  </Tab>

  <Tab title="Prod">
    **Prod Deployment** (`prod-tenant`, Live)

    ```bash theme={null}
    # 1. Ensure prod-tenant is active
    orchestrate env activate prod-tenant

    # 2. Verify flows and agent are deployed

    # 3. Verify agent is tested in Draft (PreProd)

    # 4. Deploy agent to Live (Prod) - promotes from Draft to Live
    orchestrate agents deploy \
      --name employee_onboarding

    # Note: Requires additional approvals (CAB, change ticket)
    # Note: knowledge-bases do not differentiate between draft and live environments.
    # A knowledge-base update applies to both Draft and Live simultaneously.
    ```
  </Tab>
</Tabs>

### Complete Deployment Script Example

```bash theme={null}
#!/bin/bash
set -e

# Deployment script for a specific environment
ENVIRONMENT=$1  # dev, qa, preprod, or prod
BRANCH=$(git branch --show-current)

echo "Deploying branch: $BRANCH to environment: $ENVIRONMENT"

# Determine tenant based on environment
if [ "$ENVIRONMENT" = "dev" ] || [ "$ENVIRONMENT" = "qa" ]; then
  TENANT="dev-qa-tenant"
elif [ "$ENVIRONMENT" = "preprod" ] || [ "$ENVIRONMENT" = "prod" ]; then
  TENANT="prod-tenant"
else
  echo "Invalid environment: $ENVIRONMENT"
  exit 1
fi

echo "Activating tenant: $TENANT"
orchestrate env activate $TENANT

# Deploy all knowledge-bases (affects both Draft and Live)
echo "Deploying knowledge-bases (affects both Draft and Live)..."
for kb_file in knowledge-bases/*.yaml; do
  if [ -f "$kb_file" ]; then
    kb_name=$(basename "$kb_file")
    echo "Deploying knowledge-base: $kb_name"
    orchestrate knowledge-bases import -f "$kb_file"
  fi
done

# Deploy all flows from tools/ directory (other tools auto-deploy via agents)
echo "Deploying flows..."
for tool_dir in tools/*/; do
  tool_name=$(basename "$tool_dir")
  if [ -f "$tool_dir/workflow.py" ]; then
    echo "Deploying flow: $tool_name"
    orchestrate tools import -k flow -f "$tool_dir/workflow.py"
  fi
done

# Deploy all agents (tools auto-deploy based on agent's tools: section)
echo "Deploying agents..."
for agent_dir in agents/*/; do
  agent_name=$(basename "$agent_dir")
  if [ -f "$agent_dir/agent.yaml" ]; then
    echo "Deploying agent: $agent_name"
    if [ "$ENVIRONMENT" = "dev" ] || [ "$ENVIRONMENT" = "preprod" ]; then
      # Deploy to Draft environment
      orchestrate agents import -f "$agent_dir/agent.yaml"
    elif [ "$ENVIRONMENT" = "qa" ] || [ "$ENVIRONMENT" = "prod" ]; then
      # Promote from Draft to Live
      orchestrate agents deploy --name "$agent_name"
    fi
  fi
done

echo "Note: Tools are automatically deployed when agents reference them in tools: section"

echo "Deployment to $ENVIRONMENT ($TENANT) completed successfully!"
```

### Usage

```bash theme={null}
# Deploy develop branch to dev environment
./deploy.sh dev

# Deploy qa-1.2.0 branch to qa environment
./deploy.sh qa

# Deploy staging-1.2.0 branch to staging environment
./deploy.sh staging

# Deploy main branch to prod environment
./deploy.sh prod
```

### Key Differences by Tenant

| Aspect           | dev-qa-tenant                            | prod-tenant                            |
| ---------------- | ---------------------------------------- | -------------------------------------- |
| **Activation**   | `orchestrate env activate dev-qa-tenant` | `orchestrate env activate prod-tenant` |
| **Environments** | Draft (Dev), Live (QA)                   | Draft (PreProd), Live (Prod)           |
| **Credentials**  | Dev/QA API keys                          | Prod API keys                          |
| **Connections**  | `connections/dev-qa-tenant/`             | `connections/prod-tenant/`             |
| **Approval**     | Automated                                | Requires CAB approval                  |

### Additional CLI Commands

<Tabs>
  <Tab title="List agents">
    ```bash theme={null}
    orchestrate agents list
    ```
  </Tab>

  <Tab title="Export agent">
    ```bash theme={null}
    orchestrate agents export -n employee_onboarding -k native -o agent-backup.yaml
    ```
  </Tab>

  <Tab title="Delete agent">
    ```bash theme={null}
    orchestrate agents delete -n employee_onboarding
    ```
  </Tab>

  <Tab title="List tools">
    ```bash theme={null}
    orchestrate tools list
    ```
  </Tab>

  <Tab title="Export tool">
    ```bash theme={null}
    orchestrate tools export -n hr_api -o hr-api-backup.zip
    ```
  </Tab>
</Tabs>

For complete CLI documentation, see the [watsonx Orchestrate developer documentation](https://developer.watson-orchestrate.ibm.com/).

## 11) Summary

### Core Principles

* Apply **GitOps principles** by using **Git branches** for versioning instead of manual version directories
* Replace Argo CD with a **Jenkins multi-branch pipeline** as the orchestrator
* Use the **wxO CLI** to deploy agents, flows, tools, and knowledge bases through `orchestrate` commands
* Keep agent YAML files **hand-written** and stored in Git
* Treat knowledge bases as **tenant-scoped** and not environment-scoped
* Maintain **secret-free Git** through `connections.template.yaml` plus secret substitution at deploy time
* Achieve **traceable promotions through Git merges**, safe rollbacks through Git reverts, and consistent deployments across environments without direct cluster access

### Git-Based Versioning

* **Git branches** represent environment state such as `dev`, `qa`, `preprod`, and `prod`
* **Git tags** mark deployments such as `v1.2.3-dev` and `v1.2.3-qa`
* **Git merges** promote code between environments
* **No manual version management** is required because Git handles versioning
* **Natural evolution** is possible because agent structures can change between environments

### Two-Tenant Architecture

<Tabs>
  <Tab title="Tenant 1">
    **`dev-qa-tenant`** for non-production workloads

    * Draft → Dev (`develop`)
    * Live → QA (`qa-<version>`)
  </Tab>

  <Tab title="Tenant 2">
    **`prod-tenant`** for production workloads

    * Draft → Staging (`staging-<version>`)
    * Live → Production (`main`)
  </Tab>
</Tabs>

### Key Benefits

<CardGroup cols={2}>
  <Card title="Complete Isolation" icon="shield-halved">
    Production infrastructure remains separated from development and testing
  </Card>

  <Card title="Security" icon="key">
    Different credentials, RBAC, and network policies per tenant
  </Card>

  <Card title="Compliance" icon="clipboard-check">
    Separate audit trails for production and non-production
  </Card>

  <Card title="Risk Mitigation" icon="triangle-exclamation">
    Issues in Dev or QA cannot impact PreProd or Prod
  </Card>

  <Card title="Flexibility" icon="sliders">
    Each tenant can have different configurations, quotas, and policies
  </Card>

  <Card title="Simplified Versioning" icon="tag">
    Git branches and tags replace manual version directories
  </Card>

  <Card title="Natural Evolution" icon="arrows-rotate">
    Code structure can evolve through branches
  </Card>

  <Card title="Easy Comparison" icon="code-compare">
    `git diff` shows exactly what will be promoted
  </Card>
</CardGroup>

### Promotion Path

```text theme={null}
develop → dev-qa-tenant Draft
  ↓ (PR: develop → qa-<version>)
qa-<version> → dev-qa-tenant Live
  ↓ (PR: qa-<version> → staging-<version>) ← CROSS-TENANT BOUNDARY
staging-<version> → prod-tenant Draft
  ↓ (PR: staging-<version> → main)
main → prod-tenant Live (PRODUCTION)

Git Tags:
v1.2.3-dev → v1.2.3-qa → v1.2.3-staging → v1.2.3-prod
```

### Critical Cross-Tenant Transition

The **`qa-<version> → staging-<version>`** merge is the critical boundary where:

* The pipeline switches from `dev-qa-tenant` authentication to `prod-tenant`
* API endpoints and credentials change
* Configuration files such as `tenant.yaml` and `values.yaml` may require conflict resolution
* Additional approval gates apply
* Production-grade validation begins

### Implementation Checklist

* [ ] Provision two wxO tenants: `dev-qa-tenant` and `prod-tenant`
* [ ] Set up Git repository with protected branches: `develop`, `qa-*`, `staging-*`, and `main`
* [ ] Configure branch protection rules and approval requirements
* [ ] Set up tenant-specific secrets in the secret manager
* [ ] Configure Jenkins multi-branch pipeline
* [ ] Configure Jenkins with tenant-aware credentials
* [ ] Create tenant-specific connection templates
* [ ] Implement approval gates for cross-tenant promotions
* [ ] Set up Git tagging automation in Jenkins
* [ ] Set up monitoring and alerting per tenant
* [ ] Document merge conflict resolution strategy
* [ ] Document runbooks for each tenant
* [ ] Train the team on the versioned branch workflow
* [ ] Train the team on rollback procedures