With the ADK, you can onboard custom LLM models from external providers and create model policies to manage multi-model workflows in your agents. This is referred to as the AI gateway. In the following sections, you’ll learn how to integrate new LLM models and define model policies in watsonx Orchestrate, enabling you to build more powerful agents quickly.

​
Adding LLM models

1

Most providers require an api_key value to authenticate with the LLM service. For that, you can use a connection to store sensitive data:
BASH
orchestrate connections add -a my_creds
orchestrate connections configure -a my_creds --env draft -k key_value -t team
orchestrate connections set-credentials -a my_creds --env draft -e "api_key=my_api_key"
2

Defining a provider configuration

You must also provide a JSON string with the provider configuration, like the following example:
JSON
{
  "custom_host": "https://example.com/v1/api",
  "request_timeout": 500,
}
Each provider supports different provider configurations. You can see a list of provider configurations as well as full examples of how to use these models in Examples using the supported providers.Alternatively, you can also create a model specification file, which contains all the details of your model and the provider configuration:
meta-llama-3-2-90b-vision-instruct.yaml
spec_version: v1
kind: model
name: watsonx/meta-llama/llama-3-2-90b-vision-instruct
display_name: Llama 3.2 Vision Instruct # Optional
description: Meta's Llama 3.2 Vision Instruct with 90b parameters running on WatsonX AI # Optional
tags: #Optional
  - meta
  - llama
model_type: chat # Optional. Default is "chat". Options: ["chat"|"chat_vision"|"completion"|"embedding"]
provider_config:
  watsonx_space_id: my_wxai_space_id
Although you can specify the API key directly in the provider configuration, this approach is not recommended if security is a priority.
3

Adding your model

There are two ways to add your model with the ADK:
  1. Recommended: You can import the model from the model specification file: 
    orchestrate models import -f "meta-llama-3-2-90b-vision-instruct.yaml" --app-id my_creds
  2. You can add the model by passing the JSON string and model information directly into the CLI: 
    orchestrate models add -n "watsonx/meta-llama/llama-3-2-90b-vision-instruct" -d "Meta's Llama 3.2 Vision Instruct with 90b parameters running on WatsonX AI" --display_name "Llama 3.2 Vision Instruct" --provider-config '{ "watsonx_space_id": "my_wxai_space_id" }' --app-id my_creds

​
Adding model policies

1

Define your policy

The first thing that you need to do is to define how your models will behave.
model_policy.yaml
spec_version: v1
kind: model
name: anygem
description: Balances requests between 2 Gemini models
display_name: Any Gem
policy:
  strategy:
    mode: loadbalance
  retry:
    attempts: 1
    on_status_codes: [503]
  targets:
    - model_name: virtual-model/google/gemini-2.0-flash
      weight: 0.75   # Weights must be greater than 0 and less than or equal to 1  
    - model_name: virtual-model/google/gemini-2.0-flash-lite
      weight: 0.25
You can see a full reference for options in Model policies.
2

Import the model policy

Run the following command to import the model policy:
BASH
orchestrate models policy import --file model_policy.yaml

​
Next steps

Managing LLMs

Learn how to manage your LLMs by using the ADK’s CLI.

Model policies

Learn how to create and manage model policies.

Examples with supported providers

See how to add OpenAI, Azure, AWS Bedrock, Ollama, and many more models from different providers.

Integrate the Developer Edition with SaaS

Learn how to integrate the watsonx Orchestrate Developer Edition with your watsonx Orchestrate SaaS tenant for LLM inferencing.