Configuring model policies

Model policies allow for the coordination of multiple models to accomplish tasks like load-balancing and fallback.

Adding model policies

BASH

orchestrate models policy add --name <model_name> --model <provider1>/<model_id1> --model <provider2>/<model_id2> --strategy <strategy_type> --strategy-on-code 500 --retry-on-code 503 --retry-attempts 3

Show command flags

--name / -n

string

The name of the policy you want to add.

--description / -d

string

An optional description to appear alongside the policy in the list view.

--display-name

string

An optional display name for the policy in the UI.

--strategy / -s

string

The policy mode you want to use.
loadbalance: Distributes requests between models based on weight values (default is 1 for each).
fallback: Uses another model if one becomes unavailable.
single: Uses only one model, but allows for --retry-on-code and --retry-attempts.

--strategy-on-code

list[int]

A list of HTTP error codes which triggers the strategy. Used for fallback strategy.

--retry-on-code

list[int]

A list of HTTP error codes for which the model should retry the request.

--retry-attempts

int

How many attempts it should make before stopping.

Importing model policies

BASH

orchestrate models policy import --file my_spec.yaml

After you create a model policy, assign the policy to your agent by using the ADK and the agent YAML file. Assign the policy the same way that you assign a regular model. You cannot assign a model policy to a Generative Prompt Activity in a flow.

Where the my_spec.yaml file follows this structure:

Load balancing
Fallback
Single

[my_spec.yaml]

spec_version: v1
kind: model
name: anygem
description: Balances requests between 2 Gemini models
display_name: Any Gem
policy:
  strategy:
    mode: loadbalance
    on_status_codes: [503, 504]
  retry:
    attempts: 1
  targets:
    - model_name: virtual-model/google/gemini-2.0-flash
      weight: 0.75   # Weights must be greater than 0 and less than or equal to 1
    - model_name: virtual-model/google/gemini-2.0-flash-lite
      weight: 0.25

[my_spec.yaml]

spec_version: v1
kind: model
name: firstgem
description: Use the first Gemini model that doesn't return 503
display_name: First Gem
policy:
  strategy:
    mode: fallback
  retry:
    attempts: 1
    on_status_codes: [503]
  targets:
    - model_name: virtual-model/google/gemini-2.0-flash
    - model_name: virtual-model/google/gemini-2.0-flash-lite

[my_spec.yaml]

    spec_version: v1
    kind: model
    name: retrygem
    description: Gemini model that retries up to 3 times on 503
    display_name: Retry Gem
    policy:
      strategy:
        mode: single
      retry:
        attempts: 3
        on_status_codes: [503]
      targets:
        - model_name: virtual-model/google/gemini-2.0-flash

Flags:

--file (-f): File path of the spec file containing the model policy configuration.

Updating model policy

Use either the add or import commands with the name of the model policy that you want to update to update the model policy.

Exporting model policy

BASH

orchestrate models policy export -n <model_name> -o <path>.zip

Show command flags

Flag	Type	Required	Description
`--name` (`-n`)	string	Yes	The model policy name to export.
`--output` (`-o`)	string	Yes	The file path where the exported data is saved.

Removing model policies

BASH

orchestrate models policy remove -n <name of policy>

Show command flags

--name / -n

string

The name of the policy you want to remove.

Release Notes

Get Started

Build

Deploy

Analyze

watsonx Orchestrate Developer Edition

watsonx Orchestrate ADK MCP Server

Reference

Legal notices

Adding model policies

Importing model policies

Updating model policy

Exporting model policy

Removing model policies

Release Notes

Get Started

Build

Deploy

Analyze

watsonx Orchestrate Developer Edition

watsonx Orchestrate ADK MCP Server

Reference

Legal notices

​Adding model policies

​Importing model policies

​Updating model policy

​Exporting model policy

​Removing model policies

Adding model policies

Importing model policies

Updating model policy

Exporting model policy

Removing model policies