Skip to main content
When you create an agent, the choice of LLM determines how effectively the agent can complete your workflow. watsonx Orchestrate provides optimized support for specific model/provider combinations, and allows registration of additional models as virtual models. [1]

​
Available Models

watsonx Orchestrate provides optimized support for the following models:
ModelDescriptionLicense
★ is optimized for high-speed inference and tool calling, designed for multilingual and system-prompt flexibility.Apache 2.0
★ is available through AWS Bedrock, optimized for high-speed inference and tool calling.Apache 2.0
★ watsonx/openai/gpt-oss-120bGPT-OSS-120b available through watsonx.ai (GovCloud only - available April mid-release)Apache 2.0
Note:
  • GPT-OSS models require special considerations. For more information, see Special considerations.
  • If you’re migrating from Llama to GPT-OSS-120B, see the Migration guide for step-by-step instructions and best practices.
  • GPT-OSS-120b is a non-IBM product governed by a third-party license that may impose use restrictions and other obligations. By using this model you agree to the terms. Read the terms.
  • GPT-OSS-120b provided by Groq is not available in AWS GovCloud.

​
Optimized Models

Models marked with a ★ have undergone extensive evaluation and are optimized for use with watsonx Orchestrate agents. These models appear as preferred when you run the orchestrate models list command, and are shown as supported on the Manage Agents page in the UI.

​
Using Groq Models Locally

For local development with the Developer Edition, you can use groq/openai/gpt-oss-120b by:
  1. Via watsonx Orchestrate proxy (default): When connected to a SaaS instance, LLM requests are automatically proxied through watsonx Orchestrate without requiring your own Groq API key
  2. With your own API key: Set GROQ_API_KEY in your .env file to use your own Groq account
This allows you to test and develop agents locally while leveraging the optimized gpt-oss-120b model.

​
Virtual Models

watsonx Orchestrate allows you to register models from additional providers as virtual models. These models can be added via the ADK and will be automatically made available to users of the platform. [2]

​
Important Considerations

  • Tool calling required: Only models that support tool calling will work with watsonx Orchestrate agents
  • Compatibility not guaranteed: Virtual models are not validated for compatibility with watsonx Orchestrate, unlike the optimized models listed above
  • Testing required: Thoroughly test virtual models before production use. See Understanding Virtual Models for detailed compatibility considerations
  • Additional costs: Using virtual models may incur additional costs from the upstream provider
For detailed information about virtual model compatibility, custom endpoints, and authentication options, see Managing custom LLMs.

​
virtual-policies

In addition to adding models, you can configure complex routing rules for your LLMs. Use virtual-policies to establish pseudo-LLM names that can load balance traffic between models or establish fallback policies for when a provider is experiencing an outage.
[1] In the Developer Edition of watsonx Orchestrate, when a WO_INSTANCE and WO_API_KEY are provided in the user’s .env file, if that instance is a SaaS instance, all LLM requests will be proxied through watsonx Orchestrate without the user needing an additional watsonx.ai entitlement.If the instance is CPD and a WO_INSTANCE and WO_API_KEY are provided, only models deployed via IFM will be available via the watsonx/ prefix.If neither of the above apply, either provide a WATSONX_SPACE_ID and WATSONX_APIKEY to a watsonx.ai account hosted in us-south, or add a virtual model running within your local Developer Edition server.[2] virtual-models and virtual-policies added to a SaaS or CPD instance of watsonx Orchestrate will not automatically be available within the Developer Edition. They will need to be manually added by the user.

​
Next steps

Migrating to GPT-OSS-120B

Step-by-step guide for migrating agents from Llama to GPT-OSS-120B with optimization tips.

Managing virtual models

Learn how to manage virtual models by using the ADK’s CLI.

Model policies

Learn how to create and manage model policies for complex LLM routing and fallback.

Examples with supported providers

See how to add OpenAI, Azure, AWS Bedrock, Ollama, and many more models from different providers.

Integrate the Developer Edition with SaaS

Learn how to integrate the watsonx Orchestrate Developer Edition with your watsonx Orchestrate SaaS tenant for LLM inferencing.