Configuring model policies

​

Adding model policies

orchestrate models policy add --name <model_name> --model <provider1>/<model_id1> --model <provider2>/<model_id2> --strategy <strategy_type> --strategy-on-code 500 --retry-on-code 503 --retry-attempts 3

• --name (-n): The name of the policy you want to add.
• --description (-d): An optional description to appear a long side the policy in the list view.
• --display-name: An optional display name for the policy in the UI
• --strategy (-s): The policy mode you want to use.
  • loadbalance: These models operate together by distributing the load of requests between them, following the distribution of weight values. By default, both weight values are attributed as 1, so the loads are evenly balanced between the models. If you want to customize the weight values, see Importing model policies.
  • fallback: If one of the models is unavailable, the agent will try to use the other one as a fallback alternative.
  • single: Uses only one model, but allows for --retry-on-code and --retry-attempts.
• --strategy-on-code: A list of HTTP error codes which triggers the strategy. Used for fallback strategy.
• --retry-on-code: A list of HTTP error codes for which the model should retry the request.
• --retry-attempts: How many attempts it should make before stopping.

​

Importing model policies

orchestrate models policy import --file my_spec.yaml

spec_version: v1
kind: model
name: anygem
description: Balances requests between 2 Gemini models
display_name: Any Gem
policy:
  strategy:
    mode: loadbalance
  retry:
    attempts: 1
    on_status_codes: [503]
  targets:
    - model_name: virtual-model/google/gemini-2.0-flash
      weight: 0.75   # Weights must be greater than 0 and less than or equal to 1
    - model_name: virtual-model/google/gemini-2.0-flash-lite
      weight: 0.25

spec_version: v1
kind: model
name: firstgem
description: Use the first gemeni model that doesn't return 503
display_name: First Gem
policy:
  strategy:
    mode: fallback
  retry:
    attempts: 1
    on_status_codes: [503]
  targets:
    - model_name: virtual-model/google/gemini-2.0-flash
    - model_name: virtual-model/google/gemini-2.0-flash-lite

    spec_version: v1
    kind: model
    name: retrygem
    description: Gemeni model that retries up to 3 times on 503
    display_name: Retry Gem
    policy:
      strategy:
        mode: single
      retry:
        attempts: 3
        on_status_codes: [503]
      targets:
        - model_name: virtual-model/google/gemini-2.0-flash

• --file (-f): File path of the spec file containing the model policy configuration.

​

Update model policy

​

Removing model policies

orchestrate models policy remove -n <name of policy>

• --name (-n): The name of the model policy that you want to remove.