Managing virtual models

Integrate third-party LLM models from a variety of supported providers as virtual models.

Supported providers

Provider	Provider ID
OpenAI	`openai`
watsonx.ai	`watsonx`
Groq	`groq`
Anthropic	`anthropic`
Google Gen AI	`google`
Gemini Enterprise Agent Platform	`vertex-ai`
Azure AI	`azure-ai`
Azure OpenAI	`azure-openai`
AWS Bedrock	`bedrock`
Mistral	`mistral-ai`
OpenRouter	`openrouter`
x.ai	`x-ai`
Ollama	`ollama`
Red Hat OpenShift AI	`redhat-ai`

When you import a model from OpenRouter, always set the max_token parameter explicitly. If you omit this parameter, the system defaults to 65536 tokens. This high token count can cause the request to fail if you do not have enough credits.
GPT-OSS-120b is a non-IBM product governed by a third-party license that may impose use restrictions and other obligations. By using this model, you agree to the terms. Read the terms.

Understanding virtual models

You can configure watsonx Orchestrate to register an external model or provider as a virtual model. Consider the following important limitations before you proceed.

Compatibility and support

Not every model and provider combination is supported or tested. The supported providers list shows providers that have been tested, but this does not guarantee that every model on every listed provider works. Consider the following:

New models and API changes: Models are released frequently, and some introduce API specification changes that can cause runtime errors.
Intermediate infrastructure: Components in the request path between watsonx Orchestrate and the provider (such as gateways, proxies, or adapters) can introduce incompatibilities or special authentication requirements that watsonx Orchestrate does not support.
Model suitability: Not all models are suited for agentic workflows. Even when a model registers successfully and runs without errors, the results might not be accurate enough for business-critical agents.

Optimized support

watsonx Orchestrate provides optimized support for:

gpt-oss-120b via Groq or AWS Bedrock
gpt-oss-120b via watsonx.ai in GovCloud environments (available in the April mid-release)

These model and provider combinations have undergone extensive testing and optimization for agent workflows.

Testing requirements

When you register virtual models with other providers, allocate sufficient time to validate that the combination works correctly. Testing can identify incompatibilities that prevent the agent from functioning as expected.

Using custom provider endpoints

If you self-host an LLM or use a model proxy or gateway as a pass-through to another provider, you can use OpenAI chat completion compatibility. This section covers authentication options for custom endpoints.

OpenAI-compatible with API key authentication

The default base URL for the openai provider is https://api.openai.com/v1. For self-hosted OpenAI-compatible endpoints, ensure that your full URL ends with /chat/completions, but exclude this path when you provide the custom_host during registration. Authentication format:

Authorization: Bearer ${apiKey}

Example:

Define the model specification

custom-openai-model.yaml

spec_version: v1
kind: model
name: openai/your-model-id
model_type: chat
provider_config:
  api_key: "your-apikey"
  custom_host: "https://your-url"

Note: Do not include /chat/completions in the custom_host value.

orchestrate models import --file custom-openai-model.yaml

OpenAI-compatible with OAuth 2.0 authentication

For self-hosted OpenAI-compatible endpoints that use OAuth 2.0 client credentials authentication, use the openai-oauth2-client-creds provider type. Use this option when your OAuth token endpoint is separate from your LLM inferencing endpoint. Prerequisites: Your LLM inferencing URL must meet the following requirements:

End with /chat/completions (per the OpenAI spec)
Accept OAuth tokens in the request header: Authorization: Bearer your-access-token

Create OAuth connection

First, create a Team application/connection in watsonx Orchestrate. See OAuth 2.0 Client Credentials for details.Example OAuth token endpoint:

response = requests.post(
    token_url,
    auth=(client_id, client_secret),
    headers={"Content-Type": "application/x-www-form-urlencoded"},
    data={"grant_type": "client_credentials"}
)

orchestrate models add \
  --name "openai-oauth2-client-creds/your-model-id" \
  --provider-config '{"custom_host": "https://your-llm-inferencing-url"}' \
  --app-id your-app-id-from-step-1

Note: Exclude /chat/completions from the custom_host value.

Custom endpoints and intermediate infrastructure such as proxies and gateways might not be fully supported. Test thoroughly before using in production.

CLI Reference

Importing from a file
Using the CLI only

Add a virtual model to watsonx Orchestrate by using the orchestrate models import command.

Define the model specification file

granite-3-3-8b-model.yaml

spec_version: v1
kind: model
name: virtual-model/watsonx/ibm/granite-3.3-8b-instruct
display_name: IBM watsonx.ai (Granite)
description: |
IBM watsonx.ai model using Space-scoped configuration.
tags:
- ibm
- watsonx
model_type: chat
provider_config:
    watsonx_space_id: my-space-id # For any non-sensitive field not already provided by the connection

Show properties

required

The schema version of the file for backwards compatibility.This value is always v1.

required

The kind of manifest file. For model definitions, this is always model.

required

The name of the model to add. This name follows the form:

virtual-model/provider (hosting platform)/name of the model

If virtual-model/ is omitted, it is automatically added and is required wherever the model is referenced, for example in an agent’s llm field.

required

The name of the model as it appears in the UI.

The description of the model as it appears in the orchestrate agents list command.

A list of tags used to quickly identify models.To set this model as the default selection in the UI dropdown on the Manage Agents page, include the special tag default.

values=[chat, embedding]

The capabilities or type of model to add.Must be one of: chat or embedding.

Configuration options required to connect to the provider. The required fields depend on the provider. See the examples below.The values in provider_config are merged with any additional configuration provided by a key_value connection bound to the model at import time.Provide secure values, or values you do not want to repeat for each model, through your connection instead.

Create an API key connection

BASH

orchestrate connections add -a watsonx_credentials
orchestrate connections configure -a watsonx_credentials --env draft -k key_value -t team
orchestrate connections set-credentials -a watsonx_credentials --env draft -e "api_key=my_watsonx_api_key"

Add the model

BASH

orchestrate models import --file watsonx-model.yaml --app-id watsonx_credentials

Arguments:

--file (-f): File path of the spec file containing the model configuration.
--app-id (-a): The app ID of a key_value connection containing provider configuration details. These will be merged with the values provided in the provider_config section of the spec.
--skip-validation: Skip the automatic post-import validation check. See Validating virtual models for details.

After a successful import, validation runs automatically for new models. Embedding models are skipped. If validation fails, the model remains registered. Decide whether to proceed with a model that does not pass all checks.

Run the orchestrate models add command to add a custom LLM to your active environment.

BASH

orchestrate models add --name watsonx/meta-llama/llama-3-2-90b-vision-instruct --app-id watsonx_ai_creds

Arguments:

--name (-n): The name of the model to add. The name must follow the pattern <provider>/<model_name>. The provider must match exactly as listed in the Supported providers section, and the model_name must match exactly the name shown in the provider’s API documentation.
--description (-d): An optional description to appear alongside the model in the list view.
--display-name: An optional display name for the model in the UI.
--provider-config: A JSON string of configuration options. These can also be provided via the connection referenced in --app-id, especially secret values. You can use the --provider-config alongside an --app-id to provide non-required values.
--type - The type of model that is being created. The supported types are:
- chat: Model that supports chat capabilities.
- embedding: Embedding model used for transforming data.
--app-id (-a): The app ID of a key_value connection containing provider configuration details. These will be merged with the values provided in --provider-config.
--skip-validation: Skip the automatic post-add validation check. See Validating virtual models for details.

After you successfully add a model, validation runs automatically. Embedding models are skipped. If validation fails, the model remains registered. Decide whether to proceed with a model that does not pass all checks.

Validating virtual models

The orchestrate models validate command tests whether a registered virtual model has the capabilities that are required for agentic behavior in watsonx Orchestrate. The command runs automatically after orchestrate models import and orchestrate models add for virtual models, and you can also run it on demand.

BASH

orchestrate models validate --name virtual-model/groq/openai/gpt-oss-120b

Arguments:

--name (-n): The name of the virtual model to validate. The name must match the registered model name exactly, including the virtual-model/ prefix.
--verbose (-v): Output full JSON results including per-test responses and tool call details.

What is tested

The command runs four test cases that cover the capabilities required by the watsonx Orchestrate agent runtime:

Test case	What it checks
`basic_invocation`	The model responds correctly to a simple prompt
`streaming`	The model supports streamed responses
`tool_calling`	The model can call a function, a tool, or both, and incorporate the result
`streaming_tool_calling`	The model can call a function, a tool, or both, and stream the response

Limitations

Virtual models only: Validation runs only against virtual models (models whose name starts with virtual-model/). You cannot use it to test out-of-the-box (OOTB) models.
Embedding models not supported: Do not use the orchestrate models validate command with embedding models. Embedding models do not support inference or tool calling and are skipped automatically during post-import validation.

Understanding the output

After the command runs, a summary table and a per-test results table are displayed. Each test shows a success or failure status and a duration in milliseconds. If any test fails, the overall result is failed. The model remains registered regardless of the outcome. Review the failing test cases to determine whether the model is suitable for your use case. The following are common failure patterns:

All tests fail with an HTTP error (for example, a 404): The model name might be incorrect, the model might not exist at the provider, or your credentials might not have access to it. Verify the model name and your connection credentials.
Inference tests pass but tool calling tests fail: The model can respond to prompts but does not support function or tool calling. Because tool calling is required for agentic behavior in watsonx Orchestrate, this model cannot be used as an agent LLM.

Validation Summary showing Result: passed and Test Results table with all four test cases at success status — All four tests passing — the model is suitable for use as an agent LLM

Validation Summary showing Result: failed and Test Results table with basic_invocation and streaming at success, tool_calling and streaming_tool_calling at failed — Partial failure — the model supports inference but not tool calling

Pass --verbose (-v) to get the full JSON output, which includes the raw model response, error codes, and tool call details for each test case:

BASH

orchestrate models validate --name virtual-model/groq/openai/gpt-oss-120b --verbose

Show Example verbose JSON output — all tests passing

{
  "model_name": "virtual-model/groq/openai/gpt-oss-120b",
  "timestamp": "2026-07-03T09:09:29.824086+00:00",
  "summary": {
    "total_tests": 4,
    "passed": 4,
    "failed": 0,
    "success_rate": 100.0,
    "total_duration_ms": 1118.34
  },
  "tests": [
    {
      "test_name": "basic_invocation",
      "status": "success",
      "message": "✓ Basic model invocation with simple prompt",
      "timestamp": "2026-07-03T09:09:28.922330+00:00",
      "duration_ms": 216.84,
      "response": "Pong!"
    },
    {
      "test_name": "streaming",
      "status": "success",
      "message": "✓ Streaming response capability",
      "timestamp": "2026-07-03T09:09:29.135084+00:00",
      "duration_ms": 212.7,
      "response": "1, 2, 3."
    },
    {
      "test_name": "tool_calling",
      "status": "success",
      "message": "✓ Function/tool calling capability",
      "timestamp": "2026-07-03T09:09:29.467501+00:00",
      "duration_ms": 332.35,
      "response": "The current weather in San Francisco is sunny with a temperature of about **72 °F**.",
      "tool_calls": [
        {
          "name": "get_weather",
          "args": { "location": "San Francisco" },
          "result": "The weather in San Francisco is sunny and 72°F"
        }
      ]
    },
    {
      "test_name": "streaming_tool_calling",
      "status": "success",
      "message": "✓ Function/tool calling capability with streamed response",
      "timestamp": "2026-07-03T09:09:29.824018+00:00",
      "duration_ms": 356.44,
      "response": "The current weather in San Francisco is sunny with a temperature of about **72 °F**.",
      "tool_calls": [
        {
          "name": "get_weather",
          "args": { "location": "San Francisco" },
          "result": "The weather in San Francisco is sunny and 72°F"
        }
      ]
    }
  ],
  "overall_status": "passed"
}

Show Example verbose JSON output — model not found (all tests failing)

{
  "model_name": "virtual-model/groq/openai/gpt-oss-121b",
  "timestamp": "2026-07-03T09:22:03.316538+00:00",
  "summary": {
    "total_tests": 4,
    "passed": 0,
    "failed": 4,
    "success_rate": 0.0,
    "total_duration_ms": 548.29
  },
  "tests": [
    {
      "test_name": "basic_invocation",
      "status": "failed",
      "message": "Error code: 404 - {'error': {'message': 'groq error: The model `openai/gpt-oss-121b` does not exist or you do not have access to it.', 'type': 'invalid_request_error', 'param': null, 'code': 'model_not_found'}, 'provider': 'groq'}",
      "timestamp": "2026-07-03T09:22:03.116113+00:00",
      "error_code": 404,
      "error_type": "NotFoundError",
      "duration_ms": 348.28
    },
    {
      "test_name": "streaming",
      "status": "failed",
      "message": "Error code: 404 - {'error': {'message': 'groq error: The model `openai/gpt-oss-121b` does not exist or you do not have access to it.', 'type': 'invalid_request_error', 'param': null, 'code': 'model_not_found'}, 'provider': 'groq'}",
      "timestamp": "2026-07-03T09:22:03.167524+00:00",
      "error_code": 404,
      "error_type": "NotFoundError",
      "duration_ms": 51.3
    },
    {
      "test_name": "tool_calling",
      "status": "failed",
      "message": "Error code: 404 - {'error': {'message': 'groq error: The model `openai/gpt-oss-121b` does not exist or you do not have access to it.', 'type': 'invalid_request_error', 'param': null, 'code': 'model_not_found'}, 'provider': 'groq'}",
      "timestamp": "2026-07-03T09:22:03.255559+00:00",
      "error_code": 404,
      "error_type": "NotFoundError",
      "duration_ms": 87.95
    },
    {
      "test_name": "streaming_tool_calling",
      "status": "failed",
      "message": "Error code: 404 - {'error': {'message': 'groq error: The model `openai/gpt-oss-121b` does not exist or you do not have access to it.', 'type': 'invalid_request_error', 'param': null, 'code': 'model_not_found'}, 'provider': 'groq'}",
      "timestamp": "2026-07-03T09:22:03.316411+00:00",
      "error_code": 404,
      "error_type": "NotFoundError",
      "duration_ms": 60.77
    }
  ],
  "overall_status": "failed"
}

Show Example verbose JSON output — model does not support tool calling (partial failure)

{
  "model_name": "virtual-model/watsonx/ibm/granite-8b-code-instruct",
  "timestamp": "2026-07-03T09:24:59.954166+00:00",
  "summary": {
    "total_tests": 4,
    "passed": 2,
    "failed": 2,
    "success_rate": 50.0,
    "total_duration_ms": 3166.54
  },
  "tests": [
    {
      "test_name": "basic_invocation",
      "status": "success",
      "message": "✓ Basic model invocation with simple prompt",
      "timestamp": "2026-07-03T09:24:57.311601+00:00",
      "duration_ms": 524.93,
      "response": "Pong"
    },
    {
      "test_name": "streaming",
      "status": "success",
      "message": "✓ Streaming response capability",
      "timestamp": "2026-07-03T09:24:58.086508+00:00",
      "duration_ms": 774.52,
      "response": "1\n2\n3\n\nIs there anything else I can help you with?"
    },
    {
      "test_name": "tool_calling",
      "status": "failed",
      "message": "Model does not support tool calling or failed to call tool",
      "timestamp": "2026-07-03T09:24:59.483736+00:00",
      "error_code": 500,
      "error_type": "ValueError",
      "duration_ms": 1396.81
    },
    {
      "test_name": "streaming_tool_calling",
      "status": "failed",
      "message": "Model does not support tool calling or failed to call tool",
      "timestamp": "2026-07-03T09:24:59.954111+00:00",
      "error_code": 500,
      "error_type": "ValueError",
      "duration_ms": 470.28
    }
  ],
  "overall_status": "failed"
}

Skipping automatic validation

Both orchestrate models import and orchestrate models add run validation automatically after a successful registration. To skip validation, pass the --skip-validation flag:

BASH

# Skip validation when importing from a spec file
orchestrate models import --file my-model.yaml --app-id my_credentials --skip-validation

# Skip validation when adding via the CLI
orchestrate models add --name virtual-model/openai/gpt-5 --app-id my_credentials --skip-validation

Skip validation in the following situations:

You are scripting bulk imports and plan to validate separately by using orchestrate models validate.
The model endpoint is temporarily unavailable at import time.
You do not want to wait for validation to complete during import.

Examples using the supported providers

The following sections provide examples and supported schemas for each model provider.

OpenAI

object

The fields which can either be set by connection or by the provider_config field of the model. Values from a connection will be merged with the provider_config.

Hide properties

string

required

The API key for the provider

string

Send requests to a custom hostname other than the default for the provider

list[string]

Add one or more additional response headers in the form ["header:value", "header2:value2"] to the request to the server.

number

The response timeout in seconds

boolean

Transforms the request to form_data.

Example usage:

Define the model specification file

Define a specification file with the model details and provider configuration:

gpt-5-2025-08-07.yaml

spec_version: v1
kind: model
name: openai/gpt-5-2025-08-07
display_name: GPT 5
description: |-
    GPT-5 is our flagship model for coding, reasoning, and agentic tasks across domains. Learn more in our GPT-5 usage guide.
tags:
- openai
- gpt
model_type: chat
provider_config:
    custom_host: https://my-openai-compatible-server

Create an API key connection

To use the OpenAI API key securely, first create a connection:

BASH

orchestrate connections add -a openai_credentials
orchestrate connections configure -a openai_credentials --env draft -k key_value -t team
orchestrate connections set-credentials -a openai_credentials --env draft -e "api_key=my_openai_key"

Add the model

Add the model by using the specification file and the connection you created:

BASH

orchestrate models import --file gpt-5-2025-08-07 --app-id openai_credentials

watsonx.ai

object

The fields which can either be set by connection or by the provider_config field of the model. Values from a connection will be merged with the provider_config.

Hide properties

string

required

The API key for the provider.

string

required

The service instance url of the watsonx.ai instance

string

conditionally required

At least one of space/project/deployment is required

string

conditionally required

At least one of space/project/deployment is required

string

conditionally required

At least one of space/project/deployment is required

string

conditionally required

When connecting to a watsonx.ai instance hosted in CPD, this is the url of the CPD cluster hosting watsonx.ai.Required connecting to on-prem (CPD) hosted wx.ai instances

string

conditionally required

When connecting to a watsonx.ai instance hosted in CPD, this is username of a user with access to the CPD cluster.Required connecting to on-prem (CPD) hosted wx.ai instances

string

conditionally required

When connecting to a watsonx.ai instance hosted in CPD, this is password of a user with access to the CPD cluster.Required connecting to on-prem (CPD) hosted wx.ai instances

number

The response timeout in seconds

Example usage:

Define the model specification file

watsonx-model.yaml

spec_version: v1
kind: model
name: watsonx/ibm/granite-3.3-8b-instruct
display_name: IBM watsonx.ai (Granite)
description: |
    IBM watsonx.ai model using Space-scoped configuration.
tags:
- ibm
- watsonx
model_type: chat
provider_config:
    watsonx_space_id: my-space-id

When registering gpt-oss-120b via watsonx.ai, you must include a config block. See Required config parameters for gpt-oss-120b for details.

Create an API key connection

BASH

orchestrate connections add -a watsonx_credentials
orchestrate connections configure -a watsonx_credentials --env draft -k key_value -t team
orchestrate connections set-credentials -a watsonx_credentials --env draft -e "api_key=my_watsonx_api_key"

When you add a watsonx.ai virtual model, include the provider configuration details. Without them, chat access to the model can fail. Provide custom host details by using the --provider-config flag in the orchestrate models add command. For more information, see Using the CLI only.

Add the model

BASH

orchestrate models import --file watsonx-model.yaml --app-id watsonx_credentials

Notes:

Provide one of: watsonx_space_id, watsonx_project_id, or watsonx_deployment_id.
Include watsonx_cpd_url, watsonx_cpd_username, watsonx_cpd_password only for on-prem (CPD) setups.
When you register Deploy on Demand (DoD) models, explicitly provide the model configuration. Set these configuration values according to the model’s requirements because they are not automatically transferred during inference from watsonx Orchestrate.
Show example
YAML
config: max_tokens: 2000 temperature: 0 decoding_method: "sample"

Groq

object

The fields which can either be set by connection or by the provider_config field of the model. Values from a connection will be merged with the provider_config.

Hide properties

string

required

The API key for the provider

string

required

Send requests to a custom hostname for the provider

list[string]

Add one or more additional response headers in the form ["header:value", "header2:value2"] to the request to the server

number

The response timeout in seconds

Example usage:

Define the model specification file

Define a specification file with the model details and provider configuration:

gpt-oss-120b.yaml

spec_version: v1
kind: model
name: virtual-model/groq/openai/gpt-oss-120b
display_name: openai/gpt-oss-120b # Optional
description: Welcome to the gpt-oss series, OpenAI's open-weight models designed for powerful reasoning, agentic tasks, and versatile developer use cases.
tags:
  - openai
  - gpt-oss-120b
model_type: chat # Optional. Default is "chat". Options: ["chat"|"embedding"]
app_id: groq_credentials
provider_config:
    custom_host: https://api.groq.com/openai/v1

When you register gpt-oss-120b as a virtual model, you must include a config block. See Required config parameters for gpt-oss-120b for details.

Create an API key connection

To use the API key securely, first create a connection:

BASH

orchestrate connections add -a groq_credentials
orchestrate connections configure -a groq_credentials --env draft -k key_value -t team
orchestrate connections set-credentials -a groq_credentials --env draft -e "api_key=my_openai_key"

Add the model

Add the model by using the specification file and the connection you created:

BASH

orchestrate models import --file gpt-oss-120b.yaml --app-id groq_credentials

Anthropic

object

The fields which can either be set by connection or by the provider_config field of the model. Values from a connection will be merged with the provider_config.

Hide properties

string

required

The API key for the provider.

string

list[string]

Add one or more additional response headers in the form ["header:value", "header2:value2"] to the request to the server.

number

The response timeout in seconds

Example usage:

Define the model specification file

anthropic-claude.yaml

spec_version: v1
kind: model
name: anthropic/claude-3
display_name: Anthropic Claude 3
description: |
    Anthropic Claude model for safe and helpful AI interactions.
tags:
- anthropic
- claude
model_type: chat
provider_config: {}

Create an API key connection

BASH

orchestrate connections add -a anthropic_credentials
orchestrate connections configure -a anthropic_credentials --env draft -k key_value -t team
orchestrate connections set-credentials -a anthropic_credentials --env draft -e "api_key=my_anthropic_key"

Add the model

BASH

orchestrate models import --file anthropic-claude.yaml --app-id anthropic_credentials

Google Gen AI

object

The fields which can either be set by connection or by the provider_config field of the model. Values from a connection will be merged with the provider_config.

Hide properties

string

required

The API key for the provider.

list[string]

Add one or more additional response headers in the form ["header:value", "header2:value2"] to the request to the server.

number

The response timeout in seconds

Example usage:

Define the model specification file

google-genai.yaml

spec_version: v1
kind: model
name: google/gemini-2.5-pro
display_name: Google Generative AI (Gemini 2.5 Pro)
description: |
    Google Generative AI model via API key authentication.
tags:
- google
- genai
model_type: chat
provider_config: {}

Create an API key connection

BASH

orchestrate connections add -a google_credentials
orchestrate connections configure -a google_credentials --env draft -k key_value -t team
orchestrate connections set-credentials -a google_credentials --env draft -e "api_key=my_google_api_key"

Add the model

BASH

orchestrate models import --file google-genai.yaml --app-id google_credentials

Gemini Enterprise Agent Platform

Known Limitations:

API key authentication is not supported. Use service account JSON authentication instead.

For more details, see Known issues and limitations.

object

The fields which can be set in the provider_config field of the model.

Hide properties

string

required

The GCP region for Gemini Enterprise Agent Platform.

object

The complete service account JSON object for authentication. This includes all necessary credentials for accessing Gemini Enterprise Agent Platform.

list[string]

Add one or more additional response headers in the form ["header:value", "header2:value2"] to the request to the server.

number

The response timeout in seconds.

Example usage:

Define the model specification file

my-provider.yaml

spec_version: v1
kind: model
name: vertex-ai/gemini-3.1-pro-preview
display_name: Gemini Enterprise Agent Platform (Gemini 3.1 Pro)
description: Gemini Enterprise Agent Platform model via service account authentication
tags:
  - google
  - vertex-ai
model_type: chat
provider_config:
  vertex_region: <region>
  vertex_service_account_json:
    type: service_account
    project_id: <project-id>
    private_key_id: <private-key-id>
    private_key: <private-key>
    client_email: <client-email>
    client_id: <client-id>
    auth_uri: <auth-uri>
    token_uri: <token-uri>
    auth_provider_x509_cert_url: <auth-provider-cert-url>
    client_x509_cert_url: <client-cert-url>
    universe_domain: <universe-domain>

Add the model

BASH

orchestrate models import --file my-provider.yaml

Azure

object

The fields which can either be set by connection or by the provider_config field of the model. Values from a connection will be merged with the provider_config.

Hide properties

string

required

The API key for the provider

string

required

string

required

string

required

string

required

string

Send requests to a custom hostname other than the default for the provider

list[string]

Add one or more additional response headers in the form ["header:value", "header2:value2"] to the request to the server.

number

The response timeout in seconds

Example usage:

Define the model specification file

azure-gpt.yaml

spec_version: v1
kind: model
name: azure/gpt-4
display_name: Azure GPT-4
description: |
    Azure-hosted GPT model for enterprise-grade AI workloads.
tags:
- azure
- gpt
model_type: chat
provider_config:
    azure_resource_name: my-resource
    azure_deployment_id: my-deployment
    azure_api_version: 2024-05-01

Create an API key connection

BASH

orchestrate connections add -a azure_credentials
orchestrate connections configure -a azure_credentials --env draft -k key_value -t team
orchestrate connections set-credentials -a azure_credentials --env draft -e "api_key=my_azure_key"

Add the model

BASH

orchestrate models import --file azure-gpt.yaml --app-id azure_credentials

Azure OpenAI

object

The fields which can either be set by connection or by the provider_config field of the model. Values from a connection will be merged with the provider_config.

Hide properties

string

required

The API key for the provider

string

required

string

required

string

required

boolean

string

list[string]

Add one or more additional response headers in the form ["header:value", "header2:value2"] to the request to the server.

number

The response timeout in seconds

Example usage:

Define the model specification file

azure-openai-gpt.yaml

spec_version: v1
kind: model
name: azure-openai/gpt-4
display_name: Azure OpenAI GPT-4
description: |
    Azure OpenAI GPT-4 model for enterprise workloads.
tags:
- azure
- openai
model_type: chat
provider_config:
    azure_resource_name: my-resource
    azure_deployment_id: my-deployment
    azure_api_version: 2024-05-01
    custom_host: <host_url>

Create an API key connection

BASH

orchestrate connections add -a azure_openai_credentials
orchestrate connections configure -a azure_openai_credentials --env draft -k key_value -t team
orchestrate connections set-credentials -a azure_openai_credentials --env draft -e "api_key=my_azure_openai_key"

Add the model

BASH

orchestrate models import --file azure-openai-gpt.yaml --app-id azure_openai_credentials

AWS Bedrock

object

The fields which can either be set by connection or by the provider_config field of the model. Values from a connection will be merged with the provider_config.

Hide properties

string

required

The API key for the provider.Either the api_key must be provided, or both the aws_secret_access_key and aws_access_key_id must be provided

string

required

The aws_secret_access_key.Either the api_key must be provided, or both the aws_secret_access_key and aws_access_key_id must be provided

string

required

The aws_access_key_id.Either the api_key must be provided, or both the aws_secret_access_key and aws_access_key_id must be provided

string

list[string]

Add one or more additional response headers in the form ["header:value", "header2:value2"] to the request to the server.

number

The response timeout in seconds

Example usage:

Define the model specification file

aws-bedrock-model.yaml

spec_version: v1
kind: model
name: bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0
display_name: AWS Bedrock Claude
description: |
    AWS Bedrock integration for foundation models like Claude.
tags:
- aws
- bedrock
model_type: chat
provider_config:
    aws_region: us-east-1

Create an API key connection

BASH

orchestrate connections add -a aws_bedrock_credentials
orchestrate connections configure -a aws_bedrock_credentials --env draft -k key_value -t team
orchestrate connections set-credentials -a aws_bedrock_credentials --env draft -e "api_key=my_aws_key"

Add the model

BASH

orchestrate models import --file aws-bedrock-model.yaml --app-id aws_bedrock_credentials

Provide either the api_key, aws_secret_access_key, or aws_access_key_id.
Provide the model name in the name field.
When you register Deploy on Demand (DoD) models, explicitly provide the model configuration. Set these configuration values according to the model’s requirements because they are not automatically transferred during inference from watsonx Orchestrate.
Show example
YAML
config: max_tokens: 2000 temperature: 0 decoding_method: "sample"

Mistral

object

The fields which can either be set by connection or by the provider_config field of the model. Values from a connection will be merged with the provider_config.

Hide properties

string

required

The API key for the provider.

boolean

list[string]

Add one or more additional response headers in the form ["header:value", "header2:value2"] to the request to the server.

number

The response timeout in seconds

Example usage:

Define the model specification file

mistral-large.yaml

spec_version: v1
kind: model
name: mistralai/mistral-7b-instruct-v0.3
display_name: Mistral 7B Instruct v0.3
description: |
    Mistral model for general-purpose reasoning and coding tasks.
tags:
- mistral
model_type: chat
provider_config:
    mistral_fim_completion: false

Create an API key connection

BASH

orchestrate connections add -a mistral_credentials
orchestrate connections configure -a mistral_credentials --env draft -k key_value -t team
orchestrate connections set-credentials -a mistral_credentials --env draft -e "api_key=my_mistral_api_key"

Add the model

BASH

orchestrate models import --file mistral-large.yaml --app-id mistral_credentials

OpenRouter

object

The fields which can either be set by connection or by the provider_config field of the model. Values from a connection will be merged with the provider_config.

Hide properties

string

required

The API key for the provider.

list[string]

Add one or more additional response headers in the form ["header:value", "header2:value2"] to the request to the server.

number

The response timeout in seconds

Example usage:

Create an API key connection

BASH

orchestrate connections add -a openrouter_credentials
orchestrate connections configure -a openrouter_credentials --env draft -k key_value -t team
orchestrate connections set-credentials -a openrouter_credentials --env draft -e "api_key=my_openrouter_api_key"

Define the model specification file

openrouter-model.yaml

spec_version: v1
kind: model
name: openrouter/openai/gpt-5
display_name: OpenRouter GPT-5 Chat
description: |
    OpenRouter model for routing requests across multiple LLM providers.
tags:
- openrouter
- gpt
model_type: chat
provider_config: {}

Add the model

BASH

orchestrate models import --file openrouter-model.yaml --app-id openrouter_credentials

x.ai

object

The fields which can either be set by connection or by the provider_config field of the model. Values from a connection will be merged with the provider_config.

Hide properties

string

required

The API key for the provider.

list[string]

Add one or more additional response headers in the form ["header:value", "header2:value2"] to the request to the server.

number

The response timeout in seconds

Example usage:

Create an API key connection

BASH

orchestrate connections add -a xai_credentials
orchestrate connections configure -a xai_credentials --env draft -k key_value -t team
orchestrate connections set-credentials -a xai_credentials --env draft -e "api_key=xai_api_key"

Define the model specification file

xai-model.yaml

spec_version: v1
kind: model
name: virtual-model/x-ai/grok
display_name: Grok
description: |
    x.ai model
tags:
- x.ai
- gpt
model_type: chat
provider_config: {}

Add the model

BASH

orchestrate models import --file xai-model.yaml --app-id xai_credentials

Ollama

object

The fields which can either be set by connection or by the provider_config field of the model. Values from a connection will be merged with the provider_config.

Hide properties

string

required

The API key for the provider

string

required

Send requests to a custom hostname other than the default for the provider

string

conditionally required

The Ollama url to fetch the list of available ollama models

list[string]

Add one or more additional response headers in the form ["header:value", "header2:value2"] to the request to the server.

number

The response timeout in seconds

boolean

Transforms the request to form_data.

Example usage:

Start ollama

On some systems, ollama runs under systemctl. Stop it before you start the Ollama server:

systemctl stop ollama

Then start the Ollama server and download the model:

ollama pull llama3.2:latest
export OLLAMA_HOST=0.0.0.0:11434
ollama serve

Get your IP address

Get your network IP address by running:

ipconfig # get the IPv4 address

hostname -I # get the first address

ipconfig getifaddr en0

Testing your connection

Before you import the model, test your connection to confirm that the watsonx Orchestrate Developer Edition server can reach the Ollama server.

Run the following curl command to test your connection, replacing 198.51.100.42 with the IP address you obtained in the previous step:

curl --request POST \
--url http://198.51.100.42:11434/v1/chat/completions \
--header 'content-type: application/json' \
--data '{
"model": "llama3.2:latest",
"messages": [
{
"content": "Hi",
"role": "user"
}
]
}'

Enter the watsonx Orchestrate Developer Edition gateway container:

docker exec -it docker-wxo-agent-gateway-1 sh

Run the curl command again from within the container shell.

If you experience connection issues with Ollama, try the following:

Wait a few minutes after starting the server before running the command.
Restart the Ollama server.
Close any VPN clients.
Reconnect to both Wi-Fi and wired Ethernet simultaneously.
Avoid switching networks during the process.
Reset the watsonx Orchestrate Developer Edition server:

orchestrate server reset

Define the model specification file

For Ollama, you do not need to create a connection or use a real API key. Use any string, such as ollama, as the API key value.Use your local network IP address as the URL. Ollama does not work if you use localhost or 0.0.0.0 in the model specification file.

ollama-llama2.yaml

spec_version: v1
kind: model
name: ollama/llama3.2:latest
display_name: Ollama LLaMA 3.2
description: |
    Ollama-hosted LLaMA 3.2 model for local or edge deployments.
tags:
- ollama
- llama2
model_type: chat
provider_config:
    api_key: ollama
    custom_host: http://198.51.100.42:11434

Remember: Replace http://198.51.100.42:11434 with the IP address that you have obtained in the previous step.

Add the model

BASH

orchestrate models import --file ollama-llama2.yaml

Red Hat AI

object

The fields which can either be set by connection or by the provider_config field of the model. Values from a connection will be merged with the provider_config.

Hide properties

string

required

The API key for the provider.

string

required

Send requests to a custom hostname for the provider.

list[string]

Add one or more additional response headers in the form ["header:value", "header2:value2"] to the request to the server.

number

The response timeout in seconds.

Example usage:

Define the model specification file

redhat-genai.yaml

spec_version: v1
kind: model
name: redhat-ai/gpt-oss-120b
display_name: RedHat AI gpt-oss-120b
description: |
    RedHat AI model via API key authentication.
tags:
- redhat-ai
- genai
model_type: chat
provider_config:
    custom_host: https://my-redhat-ai-compatible-server

Create an API key connection

To safely use the API key, you must first create a connection:

BASH

orchestrate connections add -a redhat_ai_credentials
orchestrate connections configure -a redhat_ai_credentials --env draft -k key_value -t team
orchestrate connections set-credentials -a redhat_ai_credentials --env draft -e "api_key=my_redhat_ai_api_key"

Add the model

You can now add the model using the specification file and the connection that you created:

BASH

orchestrate models import --file redhat-genai.yaml --app-id redhat_ai_credentials

Limitations:On-Prem:

For the gpt-oss-120b model, the model name must exactly match gpt-oss-120b. If a different model name is used, the ReAct Core style needs to be explicitly configured in the Builder UI.
If the provider uses custom CA certificates, follow the certificate import steps outlined in the on-prem documentation.

SaaS:

For the gpt-oss-120b model, the model name must exactly match gpt-oss-120b. If a different model name is used, the ReAct Core style needs to be explicitly configured in the Builder UI.
Supported only for endpoints using certificates issued by a publicly trusted Certificate Authority (CA).

For more information about Red Hat OpenShift AI setup and configuration, refer to the Red Hat OpenShift AI documentation.

List all LLMs

Run the orchestrate models list command to see all available LLMs in your active environment.

BASH

orchestrate models list

By default, the command displays a table of available models. To get raw output, add the --raw (-r) flag.

Removing custom LLMs

Run the orchestrate models remove command with the --name (-n) flag to specify the LLM to remove.

BASH

orchestrate models remove -n <model-name-unique-identifier-to-delete>

Exporting custom LLM

Run the orchestrate models export command to export LLMs from your active environment.

BASH

orchestrate models export -n <model_name> -o <path>.zip

Show command flags

Flag	Type	Required	Description
`--name` (`-n`)	string	Yes	The model name to export.
`--output` (`-o`)	string	Yes	The file path where the exported data is saved.

Updating custom LLM

To update a custom LLM, remove it and then add it again:

BASH

orchestrate models remove -n <model-name-unique-identifier-to-delete>
orchestrate models add --name watsonx/meta-llama/llama-3-2-90b-vision-instruct --app-id watsonx_ai_creds

Additional configuration options

Setting a default LLM in the UI

If you use an on-premises installation with models provisioned only as virtual models, you can specify which model appears as the default in the user interface. Add the default tag under the tags section of a model with the type set to chat.

granite-default-model.yaml

spec_version: v1
kind: model
name: watsonx/ibm/granite-3.3-8b-instruct
display_name: IBM watsonx.ai (Granite)
description: |
    IBM watsonx.ai model using Space-scoped configuration.
tags:
- default # <-- this marks this as the Default model in the ui dropdown
model_type: chat
provider_config:
    watsonx_space_id: my-space-id

For on-premises installations that use only externally hosted virtual models, at least one model must be set as the default. Without a default model, the Create Agent page in the UI cannot be opened.

Setting a default embedding model

If you use an on-premises installation with models provisioned only as virtual models, you can also set a default model for knowledge bases. Add the default tag under the tags section of a model with the type set to embedding.

virtual-model.yaml

spec_version: v1
kind: model
name: virtual-model/watsonx/ibm/slate-30m-english-rtrvr-v2
display_name: slate30m
tags:
  - default
model_type: embedding
provider_config:
  watsonx_space_id: xxx
  customHost: 'https://us-south.ml.cloud.ibm.com'
  api_key: xxx

Configuring LLM parameters

Configure additional LLM parameters such as temperature and seed for more control over model behavior. Set these parameters in your agent configuration:

llm_config:
  seed: 123
  temperature: 0.0

Parameters:

seed: Sets a random seed for reproducible outputs (useful for testing and debugging)
temperature: Controls randomness in responses (0.0 = deterministic, higher values = more creative)

These settings apply to the specific agent and override any default model configuration.

Release Notes

Get Started

Build

Deploy

Analyze

Developer experience

Legal notices

Supported providers

Understanding virtual models

Compatibility and support

Optimized support

Testing requirements

Using custom provider endpoints

OpenAI-compatible with API key authentication

OpenAI-compatible with OAuth 2.0 authentication

CLI Reference

Validating virtual models

What is tested

Limitations

Understanding the output

Skipping automatic validation

Examples using the supported providers

List all LLMs

Removing custom LLMs

Exporting custom LLM

Updating custom LLM

Additional configuration options

Setting a default LLM in the UI

Setting a default embedding model

Configuring LLM parameters

​Supported providers

​Understanding virtual models

​Compatibility and support

​Optimized support

​Testing requirements

​Using custom provider endpoints

​OpenAI-compatible with API key authentication

​OpenAI-compatible with OAuth 2.0 authentication

​CLI Reference

​Validating virtual models

​What is tested

​Limitations

​Understanding the output

​Skipping automatic validation

​Examples using the supported providers

​List all LLMs

​Removing custom LLMs

​Exporting custom LLM

​Updating custom LLM

​Additional configuration options

​Setting a default LLM in the UI

​Setting a default embedding model

​Configuring LLM parameters

Supported providers

Understanding virtual models

Compatibility and support

Optimized support

Testing requirements

Using custom provider endpoints

OpenAI-compatible with API key authentication

OpenAI-compatible with OAuth 2.0 authentication

CLI Reference

Validating virtual models

What is tested

Limitations

Understanding the output

Skipping automatic validation

Examples using the supported providers

List all LLMs

Removing custom LLMs

Exporting custom LLM

Updating custom LLM

Additional configuration options

Setting a default LLM in the UI

Setting a default embedding model

Configuring LLM parameters