> ## Documentation Index
> Fetch the complete documentation index at: https://developer.watson-orchestrate.ibm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Managing virtual models

Integrate third-party LLM models from a variety of [supported providers](#supported-providers) as virtual models.

## Supported providers

| Provider                                                                                              | Provider ID    |
| ----------------------------------------------------------------------------------------------------- | -------------- |
| [OpenAI](https://platform.openai.com/)                                                                | `openai`       |
| [watsonx.ai](https://www.ibm.com/products/watsonx-ai)                                                 | `watsonx`      |
| [Groq](https://groq.com)                                                                              | `groq`         |
| [Anthropic](https://docs.anthropic.com/en/home)                                                       | `anthropic`    |
| [Google Gen AI](https://ai.google.dev/)                                                               | `google`       |
| [Gemini Enterprise Agent Platform](https://cloud.google.com/products/agent-builder)                   | `vertex-ai`    |
| [Azure AI](https://azure.microsoft.com/en-us/products/ai-model-catalog)                               | `azure-ai`     |
| [Azure OpenAI](https://azure.microsoft.com/products/ai-services/openai-service)                       | `azure-openai` |
| [AWS Bedrock](https://aws.amazon.com/bedrock/)                                                        | `bedrock`      |
| [Mistral](https://admin.mistral.ai/organization)                                                      | `mistral-ai`   |
| [OpenRouter](https://openrouter.ai/)                                                                  | `openrouter`   |
| [x.ai](https://x.ai/)                                                                                 | `x-ai`         |
| [Ollama](https://ollama.com/)                                                                         | `ollama`       |
| [Red Hat OpenShift AI](https://www.redhat.com/en/technologies/cloud-computing/openshift/openshift-ai) | `redhat-ai`    |

<Note>
  * When you import a model from OpenRouter, always set the `max_token` parameter explicitly. If you omit this parameter, the system defaults to 65536 tokens. This high token count can cause the request to fail if you do not have enough credits.
  * **GPT-OSS-120b** is a non-IBM product governed by a third-party license that may impose use restrictions and other obligations. By using this model, you agree to the terms. [Read the terms](https://www.ibm.com/support/customer/csol/terms/?id=i126-9451\&lc=en).
</Note>

## Understanding virtual models

You can configure watsonx Orchestrate to register an external model or provider as a virtual model. Consider the following important limitations before you proceed.

### Compatibility and support

Not every model and provider combination is supported or tested. The [supported providers](#supported-providers) list shows providers that have been tested, but this does not guarantee that every model on every listed provider works. Consider the following:

* **New models and API changes**: Models are released frequently, and some introduce API specification changes that can cause runtime errors.
* **Intermediate infrastructure**: Components in the request path between watsonx Orchestrate and the provider (such as gateways, proxies, or adapters) can introduce incompatibilities or special authentication requirements that watsonx Orchestrate does not support.
* **Model suitability**: Not all models are suited for agentic workflows. Even when a model registers successfully and runs without errors, the results might not be accurate enough for business-critical agents.

### Optimized support

watsonx Orchestrate provides optimized support for:

* **gpt-oss-120b** via Groq or AWS Bedrock
* **gpt-oss-120b** via watsonx.ai in GovCloud environments (available in the April mid-release)

These model and provider combinations have undergone extensive testing and optimization for agent workflows.

### Testing requirements

When you register virtual models with other providers, allocate sufficient time to validate that the combination works correctly. Testing can identify incompatibilities that prevent the agent from functioning as expected.

## Using custom provider endpoints

If you self-host an LLM or use a model proxy or gateway as a pass-through to another provider, you can use OpenAI chat completion compatibility. This section covers authentication options for custom endpoints.

### OpenAI-compatible with API key authentication

The default base URL for the `openai` provider is `https://api.openai.com/v1`. For self-hosted OpenAI-compatible endpoints, ensure that your full URL ends with `/chat/completions`, but exclude this path when you provide the `custom_host` during registration.

**Authentication format**:

```
Authorization: Bearer ${apiKey}
```

**Example**:

<Steps>
  <Step title="Define the model specification">
    ```yaml custom-openai-model.yaml theme={null}
    spec_version: v1
    kind: model
    name: openai/your-model-id
    model_type: chat
    provider_config:
      api_key: "your-apikey"
      custom_host: "https://your-url"
    ```

    Note: Do not include `/chat/completions` in the `custom_host` value.
  </Step>

  <Step title="Register the model">
    ```bash theme={null}
    orchestrate models import --file custom-openai-model.yaml
    ```
  </Step>
</Steps>

### OpenAI-compatible with OAuth 2.0 authentication

For self-hosted OpenAI-compatible endpoints that use OAuth 2.0 client credentials authentication, use the `openai-oauth2-client-creds` provider type. Use this option when your OAuth token endpoint is separate from your LLM inferencing endpoint.

**Prerequisites**: Your LLM inferencing URL must meet the following requirements:

* End with `/chat/completions` (per the OpenAI spec)
* Accept OAuth tokens in the request header: `Authorization: Bearer your-access-token`

<Steps>
  <Step title="Create OAuth connection">
    First, create a Team application/connection in watsonx Orchestrate. See [OAuth 2.0 Client Credentials](../connections/build_connections#oauth-auth-client-credentials-flow) for details.

    Example OAuth token endpoint:

    ```python theme={null}
    response = requests.post(
        token_url,
        auth=(client_id, client_secret),
        headers={"Content-Type": "application/x-www-form-urlencoded"},
        data={"grant_type": "client_credentials"}
    )
    ```
  </Step>

  <Step title="Register the model">
    ```bash theme={null}
    orchestrate models add \
      --name "openai-oauth2-client-creds/your-model-id" \
      --provider-config '{"custom_host": "https://your-llm-inferencing-url"}' \
      --app-id your-app-id-from-step-1
    ```

    Note: Exclude `/chat/completions` from the `custom_host` value.
  </Step>
</Steps>

<Warning>
  Custom endpoints and intermediate infrastructure such as proxies and gateways might not be fully supported. Test thoroughly before using in production.
</Warning>

## CLI Reference

<Tabs>
  <Tab title="Importing from a file">
    Add a virtual model to watsonx Orchestrate by using the `orchestrate models import` command.

    <Steps>
      <Step title="Define the model specification file">
        ```yaml granite-3-3-8b-model.yaml theme={null}
        spec_version: v1
        kind: model
        name: virtual-model/watsonx/ibm/granite-3.3-8b-instruct
        display_name: IBM watsonx.ai (Granite)
        description: |
        IBM watsonx.ai model using Space-scoped configuration.
        tags:
        - ibm
        - watsonx
        model_type: chat
        provider_config:
            watsonx_space_id: my-space-id # For any non-sensitive field not already provided by the connection
        ```

        <Expandable title="properties">
          <ResponseField name="spec_version" required>
            The schema version of the file for backwards compatibility.

            **This value is always v1.**
          </ResponseField>

          <ResponseField name="kind" required>
            The kind of manifest file. For model definitions, this is always `model`.
          </ResponseField>

          <ResponseField name="name" required={true}>
            The name of the model to add. This name follows the form:

            ```
            virtual-model/provider (hosting platform)/name of the model
            ```

            If `virtual-model/` is omitted, it is automatically added and is required wherever the model
            is referenced, for example in an agent's `llm` field.
          </ResponseField>

          <ResponseField name="display_name" required={true}>
            The name of the model as it appears in the UI.
          </ResponseField>

          <ResponseField name="description">
            The description of the model as it appears in the `orchestrate agents list` command.
          </ResponseField>

          <ResponseField name="tags">
            A list of tags used to quickly identify models.

            **To set this model as the default selection in the UI dropdown on the Manage Agents page,
            include the special tag `default`.**
          </ResponseField>

          <ResponseField name="model_type" post={['values=[chat, embedding]']}>
            The capabilities or type of model to add.

            Must be one of: `chat` or `embedding`.
          </ResponseField>

          <ResponseField name="provider_config">
            Configuration options required to connect to the provider. The required fields depend on the provider. See the examples below.

            The values in `provider_config` are merged with any additional configuration provided by
            a `key_value` connection bound to the model at import time.

            Provide secure values, or values you do not want to repeat for each model, through your connection instead.
          </ResponseField>
        </Expandable>
      </Step>

      <Step title="Create an API key connection">
        ```bash BASH theme={null}
        orchestrate connections add -a watsonx_credentials
        orchestrate connections configure -a watsonx_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a watsonx_credentials --env draft -e "api_key=my_watsonx_api_key"
        ```
      </Step>

      <Step title="Add the model">
        ```bash BASH theme={null}
        orchestrate models import --file watsonx-model.yaml --app-id watsonx_credentials
        ```

        **Arguments**:

        * `--file` (`-f`): File path of the spec file containing the model configuration.
        * `--app-id` (`-a`):  The app ID of a `key_value` connection containing provider configuration details. These will be merged with the values provided in the `provider_config` section of the spec.
        * `--skip-validation`: Skip the automatic post-import validation check. See [Validating virtual models](#validating-virtual-models) for details.

        <Note>
          After a successful import, validation runs automatically for new models. Embedding models are skipped. If validation fails, the model remains registered. Decide whether to proceed with a model that does not pass all checks.
        </Note>
      </Step>
    </Steps>
  </Tab>

  <Tab title="Using the CLI only">
    Run the `orchestrate models add` command to add a custom LLM to your active environment.

    ```bash BASH theme={null}
    orchestrate models add --name watsonx/meta-llama/llama-3-2-90b-vision-instruct --app-id watsonx_ai_creds
    ```

    **Arguments**:

    * `--name` (`-n`): The name of the model to add. The name must follow the pattern `<provider>`/`<model_name>`. The provider must match exactly as listed in the [Supported providers](#supported-providers) section, and the `model_name` must match exactly the name shown in the provider's API documentation.
    * `--description` (`-d`): An optional description to appear alongside the model in the list view.
    * `--display-name`: An optional display name for the model in the UI.
    * `--provider-config`: A JSON string of configuration options. These can also be provided via the [connection](../connections/build_connections) referenced in `--app-id`, especially secret values. You can use the `--provider-config` alongside an `--app-id` to provide non-required values.
    * `--type` - The type of model that is being created. The supported types are:
      * `chat`: Model that supports chat capabilities.
      * `embedding`: Embedding model used for transforming data.
    * `--app-id` (`-a`): The app ID of a `key_value` [connection](../connections/build_connections) containing provider configuration details. These will be merged with the values provided in `--provider-config`.
    * `--skip-validation`: Skip the automatic post-add validation check. See [Validating virtual models](#validating-virtual-models) for details.

    <Note>
      After you successfully add a model, validation runs automatically. Embedding models are skipped. If validation fails, the model remains registered. Decide whether to proceed with a model that does not pass all checks.
    </Note>
  </Tab>
</Tabs>

### Validating virtual models

The `orchestrate models validate` command tests whether a registered virtual model has the [capabilities that are required for agentic behavior](#what-is-tested) in watsonx Orchestrate. The command runs automatically after `orchestrate models import` and `orchestrate models add` for virtual models, and you can also run it on demand.

```bash BASH theme={null}
orchestrate models validate --name virtual-model/groq/openai/gpt-oss-120b
```

**Arguments**:

* `--name` (`-n`): The name of the virtual model to validate. The name must match the registered model name exactly, including the `virtual-model/` prefix.
* `--verbose` (`-v`): Output full JSON results including per-test responses and tool call details.

#### What is tested

The command runs four test cases that cover the capabilities required by the watsonx Orchestrate agent runtime:

| Test case                | What it checks                                                             |
| ------------------------ | -------------------------------------------------------------------------- |
| `basic_invocation`       | The model responds correctly to a simple prompt                            |
| `streaming`              | The model supports streamed responses                                      |
| `tool_calling`           | The model can call a function, a tool, or both, and incorporate the result |
| `streaming_tool_calling` | The model can call a function, a tool, or both, and stream the response    |

#### Limitations

<Warning>
  * **Virtual models only**: Validation runs only against virtual models (models whose name starts with `virtual-model/`). You cannot use it to test out-of-the-box (OOTB) models.
  * **Embedding models not supported**: Do not use the `orchestrate models validate` command with embedding models. Embedding models do not support inference or tool calling and are skipped automatically during post-import validation.
</Warning>

#### Understanding the output

After the command runs, a summary table and a per-test results table are displayed. Each test shows a success or failure status and a duration in milliseconds.

If any test fails, the overall result is `failed`. The model remains registered regardless of the outcome. Review the failing test cases to determine whether the model is suitable for your use case.

The following are common failure patterns:

* **All tests fail with an HTTP error (for example, a 404)**: The model name might be incorrect, the model might not exist at the provider, or your credentials might not have access to it. Verify the model name and your connection credentials.
* **Inference tests pass but tool calling tests fail**: The model can respond to prompts but does not support function or tool calling. Because tool calling is required for agentic behavior in watsonx Orchestrate, this model cannot be used as an agent LLM.

<Frame caption="All four tests passing — the model is suitable for use as an agent LLM">
  <img src="https://mintcdn.com/ibm-2e3153bf/4lO7WdxkYgXFPCSD/images/model-validation-tool-calling-success.png?fit=max&auto=format&n=4lO7WdxkYgXFPCSD&q=85&s=7c0dbe9f1023b3a9957c2ae9a84f9379" alt="Validation Summary showing Result: passed and Test Results table with all four test cases at success status" width="798" height="362" data-path="images/model-validation-tool-calling-success.png" />
</Frame>

<Frame caption="Partial failure — the model supports inference but not tool calling">
  <img src="https://mintcdn.com/ibm-2e3153bf/4lO7WdxkYgXFPCSD/images/model-validation-tool-calling-failure.png?fit=max&auto=format&n=4lO7WdxkYgXFPCSD&q=85&s=9fbd3373258a94b9dcf2fe9eb7ce9cd1" alt="Validation Summary showing Result: failed and Test Results table with basic_invocation and streaming at success, tool_calling and streaming_tool_calling at failed" width="812" height="363" data-path="images/model-validation-tool-calling-failure.png" />
</Frame>

Pass `--verbose` (`-v`) to get the full JSON output, which includes the raw model response, error codes, and tool call details for each test case:

```bash BASH theme={null}
orchestrate models validate --name virtual-model/groq/openai/gpt-oss-120b --verbose
```

<Expandable title="Example verbose JSON output — all tests passing">
  ```json theme={null}
  {
    "model_name": "virtual-model/groq/openai/gpt-oss-120b",
    "timestamp": "2026-07-03T09:09:29.824086+00:00",
    "summary": {
      "total_tests": 4,
      "passed": 4,
      "failed": 0,
      "success_rate": 100.0,
      "total_duration_ms": 1118.34
    },
    "tests": [
      {
        "test_name": "basic_invocation",
        "status": "success",
        "message": "✓ Basic model invocation with simple prompt",
        "timestamp": "2026-07-03T09:09:28.922330+00:00",
        "duration_ms": 216.84,
        "response": "Pong!"
      },
      {
        "test_name": "streaming",
        "status": "success",
        "message": "✓ Streaming response capability",
        "timestamp": "2026-07-03T09:09:29.135084+00:00",
        "duration_ms": 212.7,
        "response": "1, 2, 3."
      },
      {
        "test_name": "tool_calling",
        "status": "success",
        "message": "✓ Function/tool calling capability",
        "timestamp": "2026-07-03T09:09:29.467501+00:00",
        "duration_ms": 332.35,
        "response": "The current weather in San Francisco is sunny with a temperature of about **72 °F**.",
        "tool_calls": [
          {
            "name": "get_weather",
            "args": { "location": "San Francisco" },
            "result": "The weather in San Francisco is sunny and 72°F"
          }
        ]
      },
      {
        "test_name": "streaming_tool_calling",
        "status": "success",
        "message": "✓ Function/tool calling capability with streamed response",
        "timestamp": "2026-07-03T09:09:29.824018+00:00",
        "duration_ms": 356.44,
        "response": "The current weather in San Francisco is sunny with a temperature of about **72 °F**.",
        "tool_calls": [
          {
            "name": "get_weather",
            "args": { "location": "San Francisco" },
            "result": "The weather in San Francisco is sunny and 72°F"
          }
        ]
      }
    ],
    "overall_status": "passed"
  }
  ```
</Expandable>

<Expandable title="Example verbose JSON output — model not found (all tests failing)">
  ```json theme={null}
  {
    "model_name": "virtual-model/groq/openai/gpt-oss-121b",
    "timestamp": "2026-07-03T09:22:03.316538+00:00",
    "summary": {
      "total_tests": 4,
      "passed": 0,
      "failed": 4,
      "success_rate": 0.0,
      "total_duration_ms": 548.29
    },
    "tests": [
      {
        "test_name": "basic_invocation",
        "status": "failed",
        "message": "Error code: 404 - {'error': {'message': 'groq error: The model `openai/gpt-oss-121b` does not exist or you do not have access to it.', 'type': 'invalid_request_error', 'param': null, 'code': 'model_not_found'}, 'provider': 'groq'}",
        "timestamp": "2026-07-03T09:22:03.116113+00:00",
        "error_code": 404,
        "error_type": "NotFoundError",
        "duration_ms": 348.28
      },
      {
        "test_name": "streaming",
        "status": "failed",
        "message": "Error code: 404 - {'error': {'message': 'groq error: The model `openai/gpt-oss-121b` does not exist or you do not have access to it.', 'type': 'invalid_request_error', 'param': null, 'code': 'model_not_found'}, 'provider': 'groq'}",
        "timestamp": "2026-07-03T09:22:03.167524+00:00",
        "error_code": 404,
        "error_type": "NotFoundError",
        "duration_ms": 51.3
      },
      {
        "test_name": "tool_calling",
        "status": "failed",
        "message": "Error code: 404 - {'error': {'message': 'groq error: The model `openai/gpt-oss-121b` does not exist or you do not have access to it.', 'type': 'invalid_request_error', 'param': null, 'code': 'model_not_found'}, 'provider': 'groq'}",
        "timestamp": "2026-07-03T09:22:03.255559+00:00",
        "error_code": 404,
        "error_type": "NotFoundError",
        "duration_ms": 87.95
      },
      {
        "test_name": "streaming_tool_calling",
        "status": "failed",
        "message": "Error code: 404 - {'error': {'message': 'groq error: The model `openai/gpt-oss-121b` does not exist or you do not have access to it.', 'type': 'invalid_request_error', 'param': null, 'code': 'model_not_found'}, 'provider': 'groq'}",
        "timestamp": "2026-07-03T09:22:03.316411+00:00",
        "error_code": 404,
        "error_type": "NotFoundError",
        "duration_ms": 60.77
      }
    ],
    "overall_status": "failed"
  }
  ```
</Expandable>

<Expandable title="Example verbose JSON output — model does not support tool calling (partial failure)">
  ```json theme={null}
  {
    "model_name": "virtual-model/watsonx/ibm/granite-8b-code-instruct",
    "timestamp": "2026-07-03T09:24:59.954166+00:00",
    "summary": {
      "total_tests": 4,
      "passed": 2,
      "failed": 2,
      "success_rate": 50.0,
      "total_duration_ms": 3166.54
    },
    "tests": [
      {
        "test_name": "basic_invocation",
        "status": "success",
        "message": "✓ Basic model invocation with simple prompt",
        "timestamp": "2026-07-03T09:24:57.311601+00:00",
        "duration_ms": 524.93,
        "response": "Pong"
      },
      {
        "test_name": "streaming",
        "status": "success",
        "message": "✓ Streaming response capability",
        "timestamp": "2026-07-03T09:24:58.086508+00:00",
        "duration_ms": 774.52,
        "response": "1\n2\n3\n\nIs there anything else I can help you with?"
      },
      {
        "test_name": "tool_calling",
        "status": "failed",
        "message": "Model does not support tool calling or failed to call tool",
        "timestamp": "2026-07-03T09:24:59.483736+00:00",
        "error_code": 500,
        "error_type": "ValueError",
        "duration_ms": 1396.81
      },
      {
        "test_name": "streaming_tool_calling",
        "status": "failed",
        "message": "Model does not support tool calling or failed to call tool",
        "timestamp": "2026-07-03T09:24:59.954111+00:00",
        "error_code": 500,
        "error_type": "ValueError",
        "duration_ms": 470.28
      }
    ],
    "overall_status": "failed"
  }
  ```
</Expandable>

#### Skipping automatic validation

Both `orchestrate models import` and `orchestrate models add` run validation automatically after a successful registration. To skip validation, pass the `--skip-validation` flag:

```bash BASH theme={null}
# Skip validation when importing from a spec file
orchestrate models import --file my-model.yaml --app-id my_credentials --skip-validation

# Skip validation when adding via the CLI
orchestrate models add --name virtual-model/openai/gpt-5 --app-id my_credentials --skip-validation
```

Skip validation in the following situations:

* You are scripting bulk imports and plan to validate separately by using `orchestrate models validate`.
* The model endpoint is temporarily unavailable at import time.
* You do not want to wait for validation to complete during import.

### Examples using the supported providers

The following sections provide examples and supported schemas for each model provider.

<AccordionGroup>
  <Accordion title="OpenAI">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider
        </ResponseField>

        <ResponseField name="custom_host" type="string">
          Send requests to a custom hostname other than the default for the provider
        </ResponseField>

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>

        <ResponseField name="transform_to_form_data" type="boolean">
          Transforms the request to `form_data`.
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        Define a specification file with the model details and provider configuration:

        ```yaml gpt-5-2025-08-07.yaml theme={null}
        spec_version: v1
        kind: model
        name: openai/gpt-5-2025-08-07
        display_name: GPT 5
        description: |-
            GPT-5 is our flagship model for coding, reasoning, and agentic tasks across domains. Learn more in our GPT-5 usage guide.
        tags:
        - openai
        - gpt
        model_type: chat
        provider_config:
            custom_host: https://my-openai-compatible-server
        ```
      </Step>

      <Step title="Create an API key connection">
        To use the OpenAI API key securely, first create a connection:

        ```bash BASH theme={null}
        orchestrate connections add -a openai_credentials
        orchestrate connections configure -a openai_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a openai_credentials --env draft -e "api_key=my_openai_key"
        ```
      </Step>

      <Step title="Add the model">
        Add the model by using the specification file and the connection you created:

        ```bash BASH theme={null}
        orchestrate models import --file gpt-5-2025-08-07 --app-id openai_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="watsonx.ai">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider.
        </ResponseField>

        <ResponseField name="custom_host" type="string" required="true">
          The service instance url of the watsonx.ai instance
        </ResponseField>

        <ResponseField name="watsonx_space_id" type="string" post={['conditionally required']}>
          **At least one of space/project/deployment is required**
        </ResponseField>

        <ResponseField name="watsonx_project_id" type="string" post={['conditionally required']}>
          **At least one of space/project/deployment is required**
        </ResponseField>

        <ResponseField name="watsonx_deployment_id" type="string" post={['conditionally required']}>
          **At least one of space/project/deployment is required**
        </ResponseField>

        <ResponseField name="watsonx_cpd_url" type="string" post={['conditionally required']}>
          When connecting to a watsonx.ai instance hosted in CPD, this is the url of the CPD cluster hosting watsonx.ai.

          **Required connecting to on-prem (CPD) hosted wx.ai instances**
        </ResponseField>

        <ResponseField name="watsonx_cpd_username" type="string" post={['conditionally required']}>
          When connecting to a watsonx.ai instance hosted in CPD, this is username of a user with access to the CPD cluster.

          **Required connecting to on-prem (CPD) hosted wx.ai instances**
        </ResponseField>

        <ResponseField name="watsonx_cpd_password" type="string" post={['conditionally required']}>
          When connecting to a watsonx.ai instance hosted in CPD, this is password of a user with access to the CPD cluster.

          **Required connecting to on-prem (CPD) hosted wx.ai instances**
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        ```yaml watsonx-model.yaml theme={null}
        spec_version: v1
        kind: model
        name: watsonx/ibm/granite-3.3-8b-instruct
        display_name: IBM watsonx.ai (Granite)
        description: |
            IBM watsonx.ai model using Space-scoped configuration.
        tags:
        - ibm
        - watsonx
        model_type: chat
        provider_config:
            watsonx_space_id: my-space-id
        ```

        <Note>
          When registering **gpt-oss-120b** via watsonx.ai, you must include a `config` block. See [Required config parameters for gpt-oss-120b](./migrating_to_gpt_oss#required-config-parameters-for-gpt-oss-120b) for details.
        </Note>
      </Step>

      <Step title="Create an API key connection">
        ```bash BASH theme={null}
        orchestrate connections add -a watsonx_credentials
        orchestrate connections configure -a watsonx_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a watsonx_credentials --env draft -e "api_key=my_watsonx_api_key"
        ```

        <Note>
          When you add a watsonx.ai virtual model, include the provider configuration details. Without them, chat access to the model can fail. Provide custom host details by using the `--provider-config` flag in the `orchestrate models add` command. For more information, see [Using the CLI only](#using-the-cli-only).
        </Note>
      </Step>

      <Step title="Add the model">
        ```bash BASH theme={null}
        orchestrate models import --file watsonx-model.yaml --app-id watsonx_credentials
        ```
      </Step>
    </Steps>

    <Note>
      **Notes**:

      * Provide **one** of: `watsonx_space_id`, `watsonx_project_id`, or `watsonx_deployment_id`.
      * Include `watsonx_cpd_url`, `watsonx_cpd_username`, `watsonx_cpd_password` **only for on-prem (CPD)** setups.
      * When you register Deploy on Demand (DoD) models, explicitly provide the model configuration. Set these configuration values according to the model's requirements because they are not automatically transferred during inference from watsonx Orchestrate.
              <Expandable title="example">
                ```yaml YAML theme={null}
                config:
                    max_tokens: 2000
                    temperature: 0
                    decoding_method: "sample"
                ```
              </Expandable>
    </Note>
  </Accordion>

  <Accordion title="Groq">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider
        </ResponseField>

        <ResponseField name="custom_host" type="string" required="true">
          Send requests to a custom hostname for the provider
        </ResponseField>

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        Define a specification file with the model details and provider configuration:

        ```yaml gpt-oss-120b.yaml theme={null}
        spec_version: v1
        kind: model
        name: virtual-model/groq/openai/gpt-oss-120b
        display_name: openai/gpt-oss-120b # Optional
        description: Welcome to the gpt-oss series, OpenAI's open-weight models designed for powerful reasoning, agentic tasks, and versatile developer use cases.
        tags:
          - openai
          - gpt-oss-120b
        model_type: chat # Optional. Default is "chat". Options: ["chat"|"embedding"]
        app_id: groq_credentials
        provider_config:
            custom_host: https://api.groq.com/openai/v1
        ```

        <Note>
          When you register **gpt-oss-120b** as a virtual model, you must include a `config` block. See [Required config parameters for gpt-oss-120b](./migrating_to_gpt_oss#required-config-parameters-for-gpt-oss-120b) for details.
        </Note>
      </Step>

      <Step title="Create an API key connection">
        To use the API key securely, first create a connection:

        ```bash BASH theme={null}
        orchestrate connections add -a groq_credentials
        orchestrate connections configure -a groq_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a groq_credentials --env draft -e "api_key=my_openai_key"
        ```
      </Step>

      <Step title="Add the model">
        Add the model by using the specification file and the connection you created:

        ```bash BASH theme={null}
        orchestrate models import --file gpt-oss-120b.yaml --app-id groq_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="Anthropic">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider.
        </ResponseField>

        <ResponseField name="anthropic_beta" type="string" />

        <ResponseField name="anthropic_version" type="string" />

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        ```yaml anthropic-claude.yaml theme={null}
        spec_version: v1
        kind: model
        name: anthropic/claude-3
        display_name: Anthropic Claude 3
        description: |
            Anthropic Claude model for safe and helpful AI interactions.
        tags:
        - anthropic
        - claude
        model_type: chat
        provider_config: {}
        ```
      </Step>

      <Step title="Create an API key connection">
        ```bash BASH theme={null}
        orchestrate connections add -a anthropic_credentials
        orchestrate connections configure -a anthropic_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a anthropic_credentials --env draft -e "api_key=my_anthropic_key"
        ```
      </Step>

      <Step title="Add the model">
        ```bash BASH theme={null}
        orchestrate models import --file anthropic-claude.yaml --app-id anthropic_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="Google Gen AI">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider.
        </ResponseField>

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        ```yaml google-genai.yaml theme={null}
        spec_version: v1
        kind: model
        name: google/gemini-2.5-pro
        display_name: Google Generative AI (Gemini 2.5 Pro)
        description: |
            Google Generative AI model via API key authentication.
        tags:
        - google
        - genai
        model_type: chat
        provider_config: {}
        ```
      </Step>

      <Step title="Create an API key connection">
        ```bash BASH theme={null}
        orchestrate connections add -a google_credentials
        orchestrate connections configure -a google_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a google_credentials --env draft -e "api_key=my_google_api_key"
        ```
      </Step>

      <Step title="Add the model">
        ```bash BASH theme={null}
        orchestrate models import --file google-genai.yaml --app-id google_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="Gemini Enterprise Agent Platform">
    <Note>
      **Known Limitations:**

      * API key authentication is not supported. Use service account JSON authentication instead.

      For more details, see [Known issues and limitations](../release/knownissues).
    </Note>

    <ResponseField name="provider_config" type="object">
      The fields which can be set in the `provider_config` field of the model.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="vertex_region" type="string" required="true">
          The GCP region for Gemini Enterprise Agent Platform.
        </ResponseField>

        <ResponseField name="vertex_service_account_json" type="object">
          The complete service account JSON object for authentication. This includes all necessary credentials for accessing Gemini Enterprise Agent Platform.
        </ResponseField>

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds.
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        ```yaml my-provider.yaml theme={null}
        spec_version: v1
        kind: model
        name: vertex-ai/gemini-3.1-pro-preview
        display_name: Gemini Enterprise Agent Platform (Gemini 3.1 Pro)
        description: Gemini Enterprise Agent Platform model via service account authentication
        tags:
          - google
          - vertex-ai
        model_type: chat
        provider_config:
          vertex_region: <region>
          vertex_service_account_json:
            type: service_account
            project_id: <project-id>
            private_key_id: <private-key-id>
            private_key: <private-key>
            client_email: <client-email>
            client_id: <client-id>
            auth_uri: <auth-uri>
            token_uri: <token-uri>
            auth_provider_x509_cert_url: <auth-provider-cert-url>
            client_x509_cert_url: <client-cert-url>
            universe_domain: <universe-domain>
        ```
      </Step>

      <Step title="Add the model">
        ```bash BASH theme={null}
        orchestrate models import --file my-provider.yaml
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="Azure">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider
        </ResponseField>

        <ResponseField name="azure_resource_name" type="string" required="true" />

        <ResponseField name="azure_deployment_id" type="string" required="true" />

        <ResponseField name="azure_api_version" type="string" required="true" />

        <ResponseField name="azure_model_name" type="string" required="true" />

        <ResponseField name="custom_host" type="string">
          Send requests to a custom hostname other than the default for the provider
        </ResponseField>

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        ```yaml azure-gpt.yaml theme={null}
        spec_version: v1
        kind: model
        name: azure/gpt-4
        display_name: Azure GPT-4
        description: |
            Azure-hosted GPT model for enterprise-grade AI workloads.
        tags:
        - azure
        - gpt
        model_type: chat
        provider_config:
            azure_resource_name: my-resource
            azure_deployment_id: my-deployment
            azure_api_version: 2024-05-01
        ```
      </Step>

      <Step title="Create an API key connection">
        ```bash BASH theme={null}
        orchestrate connections add -a azure_credentials
        orchestrate connections configure -a azure_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a azure_credentials --env draft -e "api_key=my_azure_key"
        ```
      </Step>

      <Step title="Add the model">
        ```bash BASH theme={null}
        orchestrate models import --file azure-gpt.yaml --app-id azure_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="Azure OpenAI">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider
        </ResponseField>

        <ResponseField name="azure_model_name" type="string" />

        <ResponseField name="azure_resource_name" type="string" required="true" />

        <ResponseField name="azure_deployment_id" type="string" required="true" />

        <ResponseField name="azure_api_version" type="string" required="true" />

        <ResponseField name="ad_auth" type="boolean" />

        <ResponseField name="azure_auth_mode" type="string" />

        <ResponseField name="azure_managed_client_id" type="string" />

        <ResponseField name="azure_entra_client_id" type="string" />

        <ResponseField name="azure_entra_client_secret" type="string" />

        <ResponseField name="azure_entra_tenant_id" type="string" />

        <ResponseField name="azure_ad_token" type="string" />

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        ```yaml azure-openai-gpt.yaml theme={null}
        spec_version: v1
        kind: model
        name: azure-openai/gpt-4
        display_name: Azure OpenAI GPT-4
        description: |
            Azure OpenAI GPT-4 model for enterprise workloads.
        tags:
        - azure
        - openai
        model_type: chat
        provider_config:
            azure_resource_name: my-resource
            azure_deployment_id: my-deployment
            azure_api_version: 2024-05-01
            custom_host: <host_url>
        ```
      </Step>

      <Step title="Create an API key connection">
        ```bash BASH theme={null}
        orchestrate connections add -a azure_openai_credentials
        orchestrate connections configure -a azure_openai_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a azure_openai_credentials --env draft -e "api_key=my_azure_openai_key"
        ```
      </Step>

      <Step title="Add the model">
        ```bash BASH theme={null}
        orchestrate models import --file azure-openai-gpt.yaml --app-id azure_openai_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="AWS Bedrock">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider.

          **Either the `api_key` must be provided, or both the `aws_secret_access_key` and `aws_access_key_id` must be provided**
        </ResponseField>

        <ResponseField name="aws_secret_access_key" type="string" required="true">
          The aws\_secret\_access\_key.

          **Either the `api_key` must be provided, or both the `aws_secret_access_key` and `aws_access_key_id` must be provided**
        </ResponseField>

        <ResponseField name="aws_access_key_id" type="string" required="true">
          The aws\_access\_key\_id.

          **Either the `api_key` must be provided, or both the `aws_secret_access_key` and `aws_access_key_id` must be provided**
        </ResponseField>

        <ResponseField name="aws_session_token" type="string" />

        <ResponseField name="aws_region" type="string" />

        <ResponseField name="aws_auth_type" type="string" />

        <ResponseField name="aws_role_arn" type="string" />

        <ResponseField name="aws_external_id" type="string" />

        <ResponseField name="aws_s3_bucket" type="string" />

        <ResponseField name="aws_s3_object_key" type="string" />

        <ResponseField name="aws_bedrock_model" type="string" />

        <ResponseField name="aws_server_side_encryption" type="string" />

        <ResponseField name="aws_server_side_encryption_kms_key_id" type="string" />

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        ```yaml aws-bedrock-model.yaml theme={null}
        spec_version: v1
        kind: model
        name: bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0
        display_name: AWS Bedrock Claude
        description: |
            AWS Bedrock integration for foundation models like Claude.
        tags:
        - aws
        - bedrock
        model_type: chat
        provider_config:
            aws_region: us-east-1
        ```
      </Step>

      <Step title="Create an API key connection">
        ```bash BASH theme={null}
        orchestrate connections add -a aws_bedrock_credentials
        orchestrate connections configure -a aws_bedrock_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a aws_bedrock_credentials --env draft -e "api_key=my_aws_key"
        ```
      </Step>

      <Step title="Add the model">
        ```bash BASH theme={null}
        orchestrate models import --file aws-bedrock-model.yaml --app-id aws_bedrock_credentials
        ```
      </Step>
    </Steps>

    <Note>
      * Provide either the `api_key`, `aws_secret_access_key`, or `aws_access_key_id`.
      * Provide the model name in the `name` field.
      * When you register Deploy on Demand (DoD) models, explicitly provide the model configuration. Set these configuration values according to the model's requirements because they are not automatically transferred during inference from watsonx Orchestrate.
              <Expandable title="example">
                ```yaml YAML theme={null}
                config:
                    max_tokens: 2000
                    temperature: 0
                    decoding_method: "sample"
                ```
              </Expandable>
    </Note>
  </Accordion>

  <Accordion title="Mistral">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider.
        </ResponseField>

        <ResponseField name="mistral_fim_completion" type="boolean" />

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        ```yaml mistral-large.yaml theme={null}
        spec_version: v1
        kind: model
        name: mistralai/mistral-7b-instruct-v0.3
        display_name: Mistral 7B Instruct v0.3
        description: |
            Mistral model for general-purpose reasoning and coding tasks.
        tags:
        - mistral
        model_type: chat
        provider_config:
            mistral_fim_completion: false
        ```
      </Step>

      <Step title="Create an API key connection">
        ```bash BASH theme={null}
        orchestrate connections add -a mistral_credentials
        orchestrate connections configure -a mistral_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a mistral_credentials --env draft -e "api_key=my_mistral_api_key"
        ```
      </Step>

      <Step title="Add the model">
        ```bash BASH theme={null}
        orchestrate models import --file mistral-large.yaml --app-id mistral_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="OpenRouter">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider.
        </ResponseField>

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Create an API key connection">
        ```bash BASH theme={null}
        orchestrate connections add -a openrouter_credentials
        orchestrate connections configure -a openrouter_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a openrouter_credentials --env draft -e "api_key=my_openrouter_api_key"
        ```
      </Step>

      <Step title="Define the model specification file">
        ```yaml openrouter-model.yaml theme={null}
        spec_version: v1
        kind: model
        name: openrouter/openai/gpt-5
        display_name: OpenRouter GPT-5 Chat
        description: |
            OpenRouter model for routing requests across multiple LLM providers.
        tags:
        - openrouter
        - gpt
        model_type: chat
        provider_config: {}
        ```
      </Step>

      <Step title="Add the model">
        ```bash BASH theme={null}
        orchestrate models import --file openrouter-model.yaml --app-id openrouter_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="x.ai">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider.
        </ResponseField>

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Create an API key connection">
        ```bash BASH theme={null}
        orchestrate connections add -a xai_credentials
        orchestrate connections configure -a xai_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a xai_credentials --env draft -e "api_key=xai_api_key"
        ```
      </Step>

      <Step title="Define the model specification file">
        ```yaml xai-model.yaml theme={null}
        spec_version: v1
        kind: model
        name: virtual-model/x-ai/grok
        display_name: Grok
        description: |
            x.ai model
        tags:
        - x.ai
        - gpt
        model_type: chat
        provider_config: {}
        ```
      </Step>

      <Step title="Add the model">
        ```bash BASH theme={null}
        orchestrate models import --file xai-model.yaml --app-id xai_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="Ollama">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider
        </ResponseField>

        <ResponseField name="custom_host" type="string" required="true">
          Send requests to a custom hostname other than the default for the provider
        </ResponseField>

        <ResponseField name="url_to_fetch" type="string" post={["conditionally required"]}>
          The Ollama url to fetch the list of available ollama models
        </ResponseField>

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>

        <ResponseField name="transform_to_form_data" type="boolean">
          Transforms the request to `form_data`.
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Start ollama">
        On some systems, `ollama` runs under `systemctl`. Stop it before you start the Ollama server:

        ```
        systemctl stop ollama
        ```

        Then start the Ollama server and download the model:

        ```
        ollama pull llama3.2:latest
        export OLLAMA_HOST=0.0.0.0:11434
        ollama serve
        ```
      </Step>

      <Step title="Get your IP address">
        Get your network IP address by running:

        <CodeGroup>
          ```powershell Windows theme={null}
          ipconfig # get the IPv4 address
          ```

          ```bash Linux theme={null}
          hostname -I # get the first address
          ```

          ```bash macOS theme={null}
          ipconfig getifaddr en0
          ```
        </CodeGroup>
      </Step>

      <Step title="Testing your connection">
        Before you import the model, test your connection to confirm that the watsonx Orchestrate Developer Edition server can reach the Ollama server.

        1. Run the following curl command to test your connection, replacing `198.51.100.42` with the IP address you obtained in the previous step:

        ```bash theme={null}
        curl --request POST \
        --url http://198.51.100.42:11434/v1/chat/completions \
        --header 'content-type: application/json' \
        --data '{
        "model": "llama3.2:latest",
        "messages": [
        {
        "content": "Hi",
        "role": "user"
        }
        ]
        }'
        ```

        1. Enter the watsonx Orchestrate Developer Edition gateway container:

        ```bash theme={null}
        docker exec -it docker-wxo-agent-gateway-1 sh
        ```

        1. Run the curl command again from within the container shell.

        <Tip>
          If you experience connection issues with Ollama, try the following:

          * Wait a few minutes after starting the server before running the command.
          * Restart the Ollama server.
          * Close any VPN clients.
          * Reconnect to both Wi-Fi and wired Ethernet simultaneously.
          * Avoid switching networks during the process.
          * Reset the watsonx Orchestrate Developer Edition server:

          ```bash theme={null}
          orchestrate server reset
          ```
        </Tip>
      </Step>

      <Step title="Define the model specification file">
        For Ollama, you do not need to create a connection or use a real API key. Use any string, such as `ollama`, as the API key value.

        Use your local network IP address as the URL. Ollama does not work if you use `localhost` or `0.0.0.0` in the model specification file.

        ```yaml ollama-llama2.yaml theme={null}
        spec_version: v1
        kind: model
        name: ollama/llama3.2:latest
        display_name: Ollama LLaMA 3.2
        description: |
            Ollama-hosted LLaMA 3.2 model for local or edge deployments.
        tags:
        - ollama
        - llama2
        model_type: chat
        provider_config:
            api_key: ollama
            custom_host: http://198.51.100.42:11434
        ```

        <Note>**Remember:** Replace `http://198.51.100.42:11434` with the IP address that you have obtained in the previous step.</Note>
      </Step>

      <Step title="Add the model">
        ```bash BASH theme={null}
        orchestrate models import --file ollama-llama2.yaml
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="Red Hat AI">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider.
        </ResponseField>

        <ResponseField name="custom_host" type="string" required="true">
          Send requests to a custom hostname for the provider.
        </ResponseField>

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds.
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        ```yaml redhat-genai.yaml theme={null}
        spec_version: v1
        kind: model
        name: redhat-ai/gpt-oss-120b
        display_name: RedHat AI gpt-oss-120b
        description: |
            RedHat AI model via API key authentication.
        tags:
        - redhat-ai
        - genai
        model_type: chat
        provider_config:
            custom_host: https://my-redhat-ai-compatible-server
        ```
      </Step>

      <Step title="Create an API key connection">
        To safely use the API key, you must first create a connection:

        ```bash BASH theme={null}
        orchestrate connections add -a redhat_ai_credentials
        orchestrate connections configure -a redhat_ai_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a redhat_ai_credentials --env draft -e "api_key=my_redhat_ai_api_key"
        ```
      </Step>

      <Step title="Add the model">
        You can now add the model using the specification file and the connection that you created:

        ```bash BASH theme={null}
        orchestrate models import --file redhat-genai.yaml --app-id redhat_ai_credentials
        ```
      </Step>
    </Steps>

    **Limitations:**

    **On-Prem:**

    1. For the `gpt-oss-120b` model, the model name must exactly match `gpt-oss-120b`. If a different model name is used, the `ReAct Core` style needs to be explicitly configured in the Builder UI.
    2. If the provider uses custom CA certificates, follow the certificate import steps outlined in the on-prem documentation.

    **SaaS:**

    1. For the `gpt-oss-120b` model, the model name must exactly match `gpt-oss-120b`. If a different model name is used, the `ReAct Core` style needs to be explicitly configured in the Builder UI.
    2. Supported only for endpoints using certificates issued by a publicly trusted Certificate Authority (CA).

    <Note>
      For more information about Red Hat OpenShift AI setup and configuration, refer to the [Red Hat OpenShift AI documentation](https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.4).
    </Note>
  </Accordion>
</AccordionGroup>

### List all LLMs

Run the `orchestrate models list` command to see all available LLMs in your active environment.

```bash BASH theme={null}
orchestrate models list
```

<Note>
  By default, the command displays a table of available models. To get raw output, add the `--raw` (`-r`) flag.
</Note>

### Removing custom LLMs

Run the `orchestrate models remove` command with the `--name` (`-n`) flag to specify the LLM to remove.

```bash BASH theme={null}
orchestrate models remove -n <model-name-unique-identifier-to-delete>
```

### Exporting custom LLM

Run the `orchestrate models export` command to export LLMs from your active environment.

```bash BASH theme={null}
orchestrate models export -n <model_name> -o <path>.zip
```

<Expandable title="command flags">
  | Flag              | Type   | Required | Description                                     |
  | ----------------- | ------ | -------- | ----------------------------------------------- |
  | `--name` (`-n`)   | string | Yes      | The model name to export.                       |
  | `--output` (`-o`) | string | Yes      | The file path where the exported data is saved. |
</Expandable>

### Updating custom LLM

To update a custom LLM, remove it and then add it again:

```bash BASH theme={null}
orchestrate models remove -n <model-name-unique-identifier-to-delete>
orchestrate models add --name watsonx/meta-llama/llama-3-2-90b-vision-instruct --app-id watsonx_ai_creds
```

### Additional configuration options

#### Setting a default LLM in the UI

If you use an on-premises installation with models provisioned only as virtual models, you can specify which model appears as the default in the user interface. Add the `default` tag under the `tags` section of a model with the type set to `chat`.

```yaml granite-default-model.yaml [expandable] theme={null}
spec_version: v1
kind: model
name: watsonx/ibm/granite-3.3-8b-instruct
display_name: IBM watsonx.ai (Granite)
description: |
    IBM watsonx.ai model using Space-scoped configuration.
tags:
- default # <-- this marks this as the Default model in the ui dropdown
model_type: chat
provider_config:
    watsonx_space_id: my-space-id
```

<Note>
  For on-premises installations that use only externally hosted virtual models, at least one model must be set as the default. Without a default model, the Create Agent page in the UI cannot be opened.
</Note>

#### Setting a default embedding model

If you use an on-premises installation with models provisioned only as virtual models, you can also set a default model for knowledge bases. Add the `default` tag under the `tags` section of a model with the type set to `embedding`.

```yaml virtual-model.yaml [expandable] theme={null}
spec_version: v1
kind: model
name: virtual-model/watsonx/ibm/slate-30m-english-rtrvr-v2
display_name: slate30m
tags:
  - default
model_type: embedding
provider_config:
  watsonx_space_id: xxx
  customHost: 'https://us-south.ml.cloud.ibm.com'
  api_key: xxx
```

#### Configuring LLM parameters

Configure additional LLM parameters such as temperature and seed for more control over model behavior. Set these parameters in your agent configuration:

```yaml theme={null}
llm_config:
  seed: 123
  temperature: 0.0
```

**Parameters:**

* `seed`: Sets a random seed for reproducible outputs (useful for testing and debugging)
* `temperature`: Controls randomness in responses (0.0 = deterministic, higher values = more creative)

These settings apply to the specific agent and override any default model configuration.