Available Models
| Model | Description |
|---|---|
| ★ | is a Groq model optimized for high-speed inference and tool calling, designed for multilingual and system-prompt flexibility. |
| ★ watsonx/meta-llama/llama-3-2-90b-vision-instruct | Llama-3-2-90b-vision-instruct is an auto-regressive language model that uses an optimized transformer architecture. |
| ★ watsonx/meta-llama/llama-3-405b-instruct | Llama-3-405b-instruct is Meta’s largest open-sourced foundation model to date, with 405 billion parameters, optimized for dialogue use cases. |
Note:
- GPT-OSS models require special considerations. For more information, see Special considerations.
- If you’re migrating from Llama to GPT-OSS-120B, see the Migration guide for step-by-step instructions and best practices.
watsonx/meta-llama/llama-3-405b-instructis not available in AWS GovCloud.- GPT-OSS-120b is a non-IBM product governed by a third-party license that may impose use restrictions and other obligations. By using this model you agree to the terms. Read the terms.
- GPT-OSS-120b provided by Groq is not available in AWS GovCloud.
Preferred Models
While all models are available, only a subset is marked as preferred. Preferred models appear with a ★ next to their name when you run theorchestrate models list command. In the UI, only preferred models are shown as supported on the Manage Agents page.
Preferred models have undergone evaluation and are optimized for use with the watsonx Orchestrate platform. On-premises models hosted through watsonx.ai Inference Foundation Models (IFM) within watsonx Orchestrate also appear in this list.
Groq models, such as groq/openai/gpt-oss-120b, can be enabled locally by setting the GROQ_API_KEY in the .env file.
AI Gateway
virtual-models
The watsonx Orchestrate platform includes the AI Gateway, which lets you expose models from your preferred provider to watsonx Orchestrate. These models can be added to watsonx Orchestrate via the ADK and will be automatically made available to all users of the watsonx Orchestrate platform. [2] While it may be possible to add any model in this way, only models that support tool calling will work with watsonx Orchestrate. Models added via the AI Gateway are not validated for compatibility with watsonx Orchestrate, unlike the preferred models from watsonx.ai. Using a virtual model may incur additional costs from the upstream provider.virtual-policies
In addition to adding models, you can configure complex routing rules for your LLMs. Use virtual-policies to establish pseudo-LLM names that can load balance traffic between models or establish fallback policies for when a provider is experiencing an outage.[1] In the Developer Edition of watsonx Orchestrate, when a
WO_INSTANCE and WO_API_KEY are provided in the user’s .env file, if that instance is a SaaS instance, all LLM requests will be proxied through watsonx Orchestrate without the user needing an additional watsonx.ai entitlement.If the instance is CPD and a WO_INSTANCE and WO_API_KEY are provided, only models deployed via IFM will be available via the watsonx/ prefix.If neither of the above apply, either provide a WATSONX_SPACE_ID and WATSONX_APIKEY to a watsonx.ai account hosted in us-south, or add a model to the AI Gateway running within your local Developer Edition server.[2] virtual-models and virtual-policies added to a SaaS or CPD instance of watsonx Orchestrate will not automatically be available within the Developer Edition. They will need to be manually added by the user.Next steps
Migrating to GPT-OSS-120B
Step-by-step guide for migrating agents from Llama to GPT-OSS-120B with optimization tips.
Managing the AI gateway
Learn how to manage your AI Gateway LLMs by using the ADK’s CLI.
Model policies
Learn how to create and manage model policies for complex LLM routing and fallback.
Examples with supported providers
See how to add OpenAI, Azure, AWS Bedrock, Ollama, and many more models from different providers.
Integrate the Developer Edition with SaaS
Learn how to integrate the watsonx Orchestrate Developer Edition with your watsonx Orchestrate SaaS tenant for LLM inferencing.