> ## Documentation Index
> Fetch the complete documentation index at: https://developer.watson-orchestrate.ibm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# wxO CI/CD Deployment Approach (Part 4)

> Summary of core principles, Git-based versioning, two-tenant architecture, and implementation checklist for wxO CI/CD

## Summary

### Core Principles

* Apply **GitOps principles** using **Git branches** for versioning instead of manual version directories.
* Replace Argo CD with **Jenkins multi-branch pipeline** as the orchestrator.
* Use **wxO CLI** to deploy agents, flows, tools, and knowledge-bases via `orchestrate` commands.
* Agent YAML files are **hand-written** (not CLI-generated) and stored in Git.
* Knowledge-bases are **tenant-scoped** (not environment-scoped) and affect both Draft and Live environments simultaneously.
* Maintain **secret-free Git** via `connections.template.yaml` + secrets manager substitution at deploy time.
* Achieve **traceable promotions via Git merges**, safe rollbacks via Git reverts, and consistent deployments across environments without direct cluster access.

### Git-Based Versioning

* **Git branches** represent environment state (dev, qa, preprod, prod)
* **Git tags** mark deployments (v1.2.3-dev, v1.2.3-qa, etc.)
* **Git merges** promote code between environments
* **No manual version management**: Git handles all versioning automatically
* **Natural evolution**: Agent structures can change significantly between environments

### Two-Tenant Architecture

<Tabs>
  <Tab title="Tenant 1">
    **`dev-qa-tenant`** for non-production workloads

    * Draft environment → Dev (`develop` branch)
    * Live environment → QA (`qa-<version>` branches)
  </Tab>

  <Tab title="Tenant 2">
    **`prod-tenant`** for production workloads

    * Draft environment → Staging (`staging-<version>` branches)
    * Live environment → Production (`main` branch)
  </Tab>
</Tabs>

### Key Benefits

<CardGroup cols={2}>
  <Card title="Complete Isolation" icon="shield-halved">
    Production infrastructure is completely separated from development/testing
  </Card>

  <Card title="Security" icon="key">
    Different credentials, RBAC, and network policies per tenant
  </Card>

  <Card title="Compliance" icon="clipboard-check">
    Separate audit trails for production vs non-production
  </Card>

  <Card title="Risk Mitigation" icon="triangle-exclamation">
    Issues in Dev/QA cannot impact PreProd/Prod
  </Card>

  <Card title="Flexibility" icon="sliders">
    Each tenant can have different configurations, quotas, and policies
  </Card>

  <Card title="Simplified Versioning" icon="tag">
    Git branches and tags replace manual version directories
  </Card>

  <Card title="Natural Evolution" icon="arrows-rotate">
    Code structure can evolve naturally through branches
  </Card>

  <Card title="Easy Comparison" icon="code-compare">
    `git diff` shows exactly what will be promoted
  </Card>
</CardGroup>

### Promotion Path (Git Branch Merging)

```text theme={null}
develop → dev-qa-tenant Draft
  ↓ (PR: develop → qa-<version>)
qa-<version> → dev-qa-tenant Live
  ↓ (PR: qa-<version> → staging-<version>) ← CROSS-TENANT BOUNDARY
staging-<version> → prod-tenant Draft
  ↓ (PR: staging-<version> → main)
main → prod-tenant Live (PRODUCTION)

Git Tags:
v1.2.3-dev → v1.2.3-qa → v1.2.3-staging → v1.2.3
```

### Critical Cross-Tenant Transition

The qa-\<version> → staging-\<version> merge is the critical boundary where:

* Git merge switches from dev-qa-tenant to prod-tenant
* Pipeline switches authentication and API endpoints
* Config files (tenant.yaml, values.yaml) may have merge conflicts (keep staging's config)
* Additional approval gates are enforced
* Production-grade validation begins

### Implementation Checklist

* [ ] Provision two wxO tenants (dev-qa-tenant, prod-tenant)
* [ ] Set up Git repository with protected branches (develop, qa-*, staging-*, main)
* [ ] Configure branch protection rules and approval requirements
* [ ] Set up tenant-specific secrets in secrets manager
* [ ] Configure Jenkins multi-branch pipeline
* [ ] Configure Jenkins with tenant-aware credentials
* [ ] Create tenant-specific connection templates
* [ ] Implement approval gates for cross-tenant promotions (qa-\* → staging-\*)
* [ ] Set up Git tagging automation in Jenkins
* [ ] Set up monitoring and alerting per tenant
* [ ] Document merge conflict resolution strategy
* [ ] Document runbooks for each tenant
* [ ] Train team on versioned branch workflow (qa-\<version>, staging-\<version>)
* [ ] Train team on rollback procedures (git revert, git reset)