Skip to main content
Model policies allow for the coordination of multiple models to accomplish tasks like load-balancing and fallback.

​
Adding model policies

BASH
orchestrate models policy add --name <model_name> --model <provider1>/<model_id1> --model <provider2>/<model_id2> --strategy <strategy_type> --strategy-on-code 500 --retry-on-code 503 --retry-attempts 3
Flags:
  • --name (-n): The name of the policy you want to add.
  • --description (-d): An optional description to appear a long side the policy in the list view.
  • --display-name: An optional display name for the policy in the UI
  • --strategy (-s): The policy mode you want to use.
    • loadbalance: These models operate together by distributing the load of requests between them, following the distribution of weight values. By default, both weight values are attributed as 1, so the loads are evenly balanced between the models. If you want to customize the weight values, see Importing model policies.
    • fallback: If one of the models is unavailable, the agent will try to use the other one as a fallback alternative.
    • single: Uses only one model, but allows for --retry-on-code and --retry-attempts.
  • --strategy-on-code: A list of HTTP error codes which triggers the strategy. Used for fallback strategy.
  • --retry-on-code: A list of HTTP error codes for which the model should retry the request.
  • --retry-attempts: How many attempts it should make before stopping.

​
Importing model policies

BASH
orchestrate models policy import --file my_spec.yaml
Where the my_spec.yaml file follows this structure:
my_spec.yaml
spec_version: v1
kind: model
name: anygem
description: Balances requests between 2 Gemini models
display_name: Any Gem
policy:
  strategy:
    mode: loadbalance
  retry:
    attempts: 1
    on_status_codes: [503]
  targets:
    - model_name: virtual-model/google/gemini-2.0-flash
      weight: 0.75   # Weights must be greater than 0 and less than or equal to 1  
    - model_name: virtual-model/google/gemini-2.0-flash-lite
      weight: 0.25
Flags:
  • --file (-f): File path of the spec file containing the model policy configuration.

​
Update model policy

Use either the add or import commands with the name of the model policy that you want to update to update the model policy.

​
Removing model policies

BASH
orchestrate models policy remove -n <name of policy>
Flags:
  • --name (-n): The name of the model policy that you want to remove.
⌘I