> ## Documentation Index
> Fetch the complete documentation index at: https://developer.watson-orchestrate.ibm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Managing custom LLMs with the AI gateway

You can integrate third-party LLM models from a variety of [supported providers](#supported-providers) through the **AI Gateway** system. The gateway also supports the configuration of routing policies, enabling use cases such as load balancing and fallback across multiple models.

## Supported providers

| Provider                                                                        | Provider ID    |
| ------------------------------------------------------------------------------- | -------------- |
| [OpenAI](https://platform.openai.com/)                                          | `openai`       |
| [watsonx.ai](https://www.ibm.com/products/watsonx-ai)                           | `watsonx`      |
| [Groq](https://groq.com)                                                        | `groq`         |
| [Anthropic](https://docs.anthropic.com/en/home)                                 | `anthropic`    |
| [Google](https://ai.google.dev/)                                                | `google`       |
| [Azure AI](https://azure.microsoft.com/en-us/products/ai-model-catalog)         | `azure-ai`     |
| [Azure OpenAI](https://azure.microsoft.com/products/ai-services/openai-service) | `azure-openai` |
| [AWS Bedrock](https://aws.amazon.com/bedrock/)                                  | `bedrock`      |
| [Mistral](https://admin.mistral.ai/organization)                                | `mistral-ai`   |
| [OpenRouter](https://openrouter.ai/)                                            | `openrouter`   |
| [x.ai](https://x.ai/)                                                           | `x-ai`         |
| [Ollama](https://ollama.com/)                                                   | `ollama`       |

<Note>
  **Note:**
  When importing a model from OpenRouter, always set the max\_token parameter explicitly. If you skip this step, the system defaults to 65536 tokens. This high token count can cause the request to fail if you don’t have enough credits.
</Note>

## CLI Reference

<Tabs>
  <Tab title="Importing from a file">
    You can add a model to the watsonx Orchestrate AI gateway using the `orchestrate models import` command.

    <Steps>
      <Step title="Define the model specification file">
        ```yaml granite-3-3-8b-model.yaml theme={null}
        spec_version: v1
        kind: model
        name: virtual-model/watsonx/ibm/granite-3.3-8b-instruct
        display_name: IBM watsonx.ai (Granite)
        description: |
        IBM watsonx.ai model using Space-scoped configuration.
        tags:
        - ibm
        - watsonx
        model_type: chat
        provider_config:
            watsonx_space_id: my-space-id # For any non-sensitive field not already provided by the connection
        ```

        <Expandable title="properties">
          <ResponseField name="spec_version" required>
            The schema version of the file for backwards compatibility.

            **This always is v1.**
          </ResponseField>

          <ResponseField name="kind" required>
            The kind of manifest file. For model definitions, this is always `model`.
          </ResponseField>

          <ResponseField name="name" required={true}>
            The name of the model to add. This name follows the form:

            ```
            virtual-model/provider (hosting platform)/name of the model
            ```

            If `virtual-model/` is omitted it is automatically added and required everywhere this model
            is referenced. For example: an agent's `llm` field.
          </ResponseField>

          <ResponseField name="display_name" required={true}>
            The name of the model as it appears in the UI.
          </ResponseField>

          <ResponseField name="description">
            The description of the model as it appears in the `orchestrate agents list` command.
          </ResponseField>

          <ResponseField name="tags">
            A list of tags used to quickly identify models.

            **To set this model as the default model selection in the ui dropdown on the manage agents page,
            please include the special tag `default`.**
          </ResponseField>

          <ResponseField name="model_type" post={['values=[chat, chat_vision, completion, embedding]']}>
            This field describes the capabilities/ type of model to add.

            Must be one of: chat, chat\_vision, completion, embedding.
          </ResponseField>

          <ResponseField name="provider_config">
            A collection of configuration options needed to connect to provider. The required provider configuration
            fields are determined by the provider. Examples can be seen below.

            The values of the `provider_config` is merged with any additional configuration provided by
            a `key_value` connection bound to the model at import time.

            Secure values and values which you do not want to repeat for each model, should
            be provided via your connection instead.
          </ResponseField>
        </Expandable>
      </Step>

      <Step title="Create an API key connection">
        ```bash theme={null}
        orchestrate connections add -a watsonx_credentials
        orchestrate connections configure -a watsonx_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a watsonx_credentials --env draft -e "api_key=my_watsonx_api_key"
        ```
      </Step>

      <Step title="Add the model">
        ```bash theme={null}
        orchestrate models import --file watsonx-model.yaml --app-id watsonx_credentials
        ```

        **Arguments**:

        * `--file` (`-f`): File path of the spec file containing the model configuration.
        * `--app-id` (`-a`):  The app ID of a `key_value` connection containing provider configuration details. These will be merged with the values provided in the `provider_config` section of the spec.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Using the CLI only">
    Run the `orchestrate models add` command to add a custom LLM to your active environment.

    ```bash BASH theme={null}
    orchestrate models add --name groq/openai/gpt-oss-120b --app-id watsonx_ai_creds
    ```

    **Arguments**:

    * `--name` (`-n`): The name of the model you want to add. This name must follow the pattern `<provider>`/`<model_name>`. The provider must be exactly as outlined in the [Supported providers](#supported-providers) section. And the `model_name` must be exactly the same as the name that appears on the provider's API documentaion.
    * `--description` (`-d`): An optional description to appear alongside the model in the list view.
    * `--display-name`: An optional display name for the model in the UI.
    * `--provider-config`: A JSON string of configuration options. These can also be provided via the [connection](../connections/build_connections) referenced in `--app-id`, especially secret values. You can use the `--provider-config` alongside an `--app-id` to provide non-required values.
    * `--type` - The type of model that is being created. The supported types are:
      * `chat`: Model that supports chat capabilities.
      * `chat_vision`: Model that supports chat and image capabilities.
      * `completion`: Model used for completion engines.
      * `embedding`: Embedding model used for transforming data.
    * `--app-id` (`-a`): The app ID of a `key_value` [connection](../connections/build_connections) containing provider configuration details. These will be merged with the values provided in `--provider-config`.
  </Tab>
</Tabs>

### Examples using the supported providers

The following sections contain examples and the supported schemas for each model provider.

<AccordionGroup>
  <Accordion title="OpenAI">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider
        </ResponseField>

        <ResponseField name="custom_host" type="string">
          Send requests to a custom hostname other than the default for the provider
        </ResponseField>

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>

        <ResponseField name="transform_to_form_data" type="boolean">
          Transforms the request to `form_data`.
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        Then, you can define a specification file to provide the details about the model and the provider configuration specifications:

        ```yaml gpt-5-2025-08-07.yaml theme={null}
        spec_version: v1
        kind: model
        name: openai/gpt-5-2025-08-07
        display_name: GPT 5
        description: |-
            GPT-5 is our flagship model for coding, reasoning, and agentic tasks across domains. Learn more in our GPT-5 usage guide.
        tags:
        - openai
        - gpt
        model_type: chat
        provider_config:
            custom_host: https://my-openai-compatible-server
        ```
      </Step>

      <Step title="Create an API key connection">
        To safely use the OpenAI API key, you must first create a connection:

        ```bash BASH theme={null}
        orchestrate connections add -a openai_credentials
        orchestrate connections configure -a openai_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a openai_credentials --env draft -e "api_key=my_openai_key"
        ```
      </Step>

      <Step title="Add the model">
        You can now add the model using the specification file and the connection that you created:

        ```bash theme={null}
        orchestrate models import --file gpt-5-2025-08-07 --app-id openai_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="watsonx.ai">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider.
        </ResponseField>

        <ResponseField name="custom_host" type="string" required="true">
          The service instance url of the watsonx.ai instance
        </ResponseField>

        <ResponseField name="watsonx_space_id" type="string" post={['conditionally required']}>
          **At least one of space/project/deployment is required**
        </ResponseField>

        <ResponseField name="watsonx_project_id" type="string" post={['conditionally required']}>
          **At least one of space/project/deployment is required**
        </ResponseField>

        <ResponseField name="watsonx_deployment_id" type="string" post={['conditionally required']}>
          **At least one of space/project/deployment is required**
        </ResponseField>

        <ResponseField name="watsonx_cpd_url" type="string" post={['conditionally required']}>
          When connecting to a watsonx.ai instance hosted in CPD, this is the url of the CPD cluster hosting watsonx.ai.

          **Required connecting to on-prem (CPD) hosted wx.ai instances**
        </ResponseField>

        <ResponseField name="watsonx_cpd_username" type="string" post={['conditionally required']}>
          When connecting to a watsonx.ai instance hosted in CPD, this is username of a user with access to the CPD cluster.

          **Required connecting to on-prem (CPD) hosted wx.ai instances**
        </ResponseField>

        <ResponseField name="watsonx_cpd_password" type="string" post={['conditionally required']}>
          When connecting to a watsonx.ai instance hosted in CPD, this is password of a user with access to the CPD cluster.

          **Required connecting to on-prem (CPD) hosted wx.ai instances**
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        ```yaml watsonx-model.yaml theme={null}
        spec_version: v1
        kind: model
        name: watsonx/ibm/granite-3.3-8b-instruct
        display_name: IBM watsonx.ai (Granite)
        description: |
            IBM watsonx.ai model using Space-scoped configuration.
        tags:
        - ibm
        - watsonx
        model_type: chat
        provider_config:
            watsonx_space_id: my-space-id
        ```
      </Step>

      <Step title="Create an API key connection">
        ```bash theme={null}
        orchestrate connections add -a watsonx_credentials
        orchestrate connections configure -a watsonx_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a watsonx_credentials --env draft -e "api_key=my_watsonx_api_key"
        ```

        <Note>
          **Note:**

          When you add a watsonx.ai virtual model, include the provider-config details. Without them, chat access to the model may fail. Provide custom host details using the --provider-config flag in the orchestrate models add command. For more information, see [Using the CLI only](#using-the-cli-only).
        </Note>
      </Step>

      <Step title="Add the model">
        ```bash theme={null}
        orchestrate models import --file watsonx-model.yaml --app-id watsonx_credentials
        ```
      </Step>
    </Steps>

    <Note>
      **Notes**:

      * Provide **one** of: `watsonx_space_id`, `watsonx_project_id`, or `watsonx_deployment_id`.
      * Include `watsonx_cpd_url`, `watsonx_cpd_username`, `watsonx_cpd_password` **only for on-prem (CPD)** setups.
      * When deploying Deploy on Demand (DoD) models, you need to explicitly provide the model configuration during registration. Set these configuration values according to the model’s requirements, since they don’t automatically transfer during inference from the watsonx Orchestrate side.
              <Expandable title="example">
                ```yaml YAML theme={null}
                config:
                    max_tokens: 2000
                    temperature: 0
                    decoding_method: "sample"
                ```
              </Expandable>
    </Note>
  </Accordion>

  <Accordion title="Groq">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider
        </ResponseField>

        <ResponseField name="custom_host" type="string" required="true">
          Send requests to a custom hostname for the provider
        </ResponseField>

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        Then, you can define a specification file to provide the details about the model and the provider configuration specifications:

        ```yaml gpt-oss-120b.yaml theme={null}
        spec_version: v1
        kind: model
        name: virtual-model/groq/openai/gpt-oss-120b
        display_name: openai/gpt-oss-120b # Optional
        description: Welcome to the gpt-oss series, OpenAI's open-weight models designed for powerful reasoning, agentic tasks, and versatile developer use cases.
        tags:
          - openai
          - gpt-oss-120b
        model_type: chat # Optional. Default is "chat". Options: ["chat"|"chat_vision"|"completion"|"embedding"]
        app_id: groq_credentials
        provider_config:
            custom_host: https://api.groq.com/openai/v1
        ```
      </Step>

      <Step title="Create an API key connection">
        To safely use the API key, you must first create a connection:

        ```bash BASH theme={null}
        orchestrate connections add -a groq_credentials
        orchestrate connections configure -a groq_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a groq_credentials --env draft -e "api_key=my_openai_key"
        ```
      </Step>

      <Step title="Add the model">
        You can now add the model using the specification file and the connection that you created:

        ```bash theme={null}
        orchestrate models import --file gpt-oss-120b.yaml --app-id groq_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="Anthropic">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider.
        </ResponseField>

        <ResponseField name="anthropic_beta" type="string" />

        <ResponseField name="anthropic_version" type="string" />

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        ```yaml anthropic-claude.yaml theme={null}
        spec_version: v1
        kind: model
        name: anthropic/claude-3
        display_name: Anthropic Claude 3
        description: |
            Anthropic Claude model for safe and helpful AI interactions.
        tags:
        - anthropic
        - claude
        model_type: chat
        provider_config: {}
        ```
      </Step>

      <Step title="Create an API key connection">
        ```bash theme={null}
        orchestrate connections add -a anthropic_credentials
        orchestrate connections configure -a anthropic_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a anthropic_credentials --env draft -e "api_key=my_anthropic_key"
        ```
      </Step>

      <Step title="Add the model">
        ```bash theme={null}
        orchestrate models import --file anthropic-claude.yaml --app-id anthropic_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="Google">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider.
        </ResponseField>

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        ```yaml google-genai.yaml theme={null}
        spec_version: v1
        kind: model
        name: google/gemini-2.5-pro
        display_name: Google Generative AI (Gemini 2.5 Pro)
        description: |
            Google Generative AI model via API key authentication.
        tags:
        - google
        - genai
        model_type: chat
        provider_config: {}
        ```
      </Step>

      <Step title="Create an API key connection">
        ```bash theme={null}
        orchestrate connections add -a google_credentials
        orchestrate connections configure -a google_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a google_credentials --env draft -e "api_key=my_google_api_key"
        ```
      </Step>

      <Step title="Add the model">
        ```bash theme={null}
        orchestrate models import --file google-genai.yaml --app-id google_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="Azure">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider
        </ResponseField>

        <ResponseField name="azure_resource_name" type="string" required="true" />

        <ResponseField name="azure_deployment_id" type="string" required="true" />

        <ResponseField name="azure_api_version" type="string" required="true" />

        <ResponseField name="azure_model_name" type="string" required="true" />

        <ResponseField name="custom_host" type="string">
          Send requests to a custom hostname other than the default for the provider
        </ResponseField>

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        ```yaml azure-gpt.yaml theme={null}
        spec_version: v1
        kind: model
        name: azure/gpt-4
        display_name: Azure GPT-4
        description: |
            Azure-hosted GPT model for enterprise-grade AI workloads.
        tags:
        - azure
        - gpt
        model_type: chat
        provider_config:
            azure_resource_name: my-resource
            azure_deployment_id: my-deployment
            azure_api_version: 2024-05-01
        ```
      </Step>

      <Step title="Create an API key connection">
        ```bash theme={null}
        orchestrate connections add -a azure_credentials
        orchestrate connections configure -a azure_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a azure_credentials --env draft -e "api_key=my_azure_key"
        ```
      </Step>

      <Step title="Add the model">
        ```bash theme={null}
        orchestrate models import --file azure-gpt.yaml --app-id azure_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="Azure OpenAI">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider
        </ResponseField>

        <ResponseField name="resource_name" type="string" required="true" />

        <ResponseField name="deployment_id" type="string" required="true" />

        <ResponseField name="api_version" type="string" required="true" />

        <ResponseField name="ad_auth" type="boolean" />

        <ResponseField name="azure_auth_mode" type="string" />

        <ResponseField name="azure_managed_client_id" type="string" />

        <ResponseField name="azure_entra_client_id" type="string" />

        <ResponseField name="azure_entra_client_secret" type="string" />

        <ResponseField name="azure_entra_tenant_id" type="string" />

        <ResponseField name="azure_ad_token" type="string" />

        <ResponseField name="azure_model_name" type="string" />

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        ```yaml azure-openai-gpt.yaml theme={null}
        spec_version: v1
        kind: model
        name: azure-openai/gpt-4
        display_name: Azure OpenAI GPT-4
        description: |
            Azure OpenAI GPT-4 model for enterprise workloads.
        tags:
        - azure
        - openai
        model_type: chat
        provider_config:
            azure_resource_name: my-resource
            azure_deployment_id: my-deployment
            azure_api_version: 2024-05-01
        ```
      </Step>

      <Step title="Create an API key connection">
        ```bash theme={null}
        orchestrate connections add -a azure_openai_credentials
        orchestrate connections configure -a azure_openai_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a azure_openai_credentials --env draft -e "api_key=my_azure_openai_key"
        ```
      </Step>

      <Step title="Add the model">
        ```bash theme={null}
        orchestrate models import --file azure-openai-gpt.yaml --app-id azure_openai_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="AWS Bedrock">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider.

          **Either the `api_key` must be provided, or both the `aws_secret_access_key` and `aws_access_key_id` must be provided**
        </ResponseField>

        <ResponseField name="aws_secret_access_key" type="string" required="true">
          The aws\_secret\_access\_key.

          **Either the `api_key` must be provided, or both the `aws_secret_access_key` and `aws_access_key_id` must be provided**
        </ResponseField>

        <ResponseField name="aws_access_key_id" type="string" required="true">
          The aws\_access\_key\_id.

          **Either the `api_key` must be provided, or both the `aws_secret_access_key` and `aws_access_key_id` must be provided**
        </ResponseField>

        <ResponseField name="aws_session_token" type="string" />

        <ResponseField name="aws_region" type="string" />

        <ResponseField name="aws_auth_type" type="string" />

        <ResponseField name="aws_role_arn" type="string" />

        <ResponseField name="aws_external_id" type="string" />

        <ResponseField name="aws_s3_bucket" type="string" />

        <ResponseField name="aws_s3_object_key" type="string" />

        <ResponseField name="aws_bedrock_model" type="string" />

        <ResponseField name="aws_server_side_encryption" type="string" />

        <ResponseField name="aws_server_side_encryption_kms_key_id" type="string" />

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        ```yaml aws-bedrock-model.yaml theme={null}
        spec_version: v1
        kind: model
        name: bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0
        display_name: AWS Bedrock Claude
        description: |
            AWS Bedrock integration for foundation models like Claude.
        tags:
        - aws
        - bedrock
        model_type: chat
        provider_config:
            aws_region: us-east-1
        ```
      </Step>

      <Step title="Create an API key connection">
        ```bash theme={null}
        orchestrate connections add -a aws_bedrock_credentials
        orchestrate connections configure -a aws_bedrock_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a aws_bedrock_credentials --env draft -e "api_key=my_aws_key"
        ```
      </Step>

      <Step title="Add the model">
        ```bash theme={null}
        orchestrate models import --file aws-bedrock-model.yaml --app-id aws_bedrock_credentials
        ```
      </Step>
    </Steps>

    <Note>
      **Note**:

      * You must provide either the `api_key`, `aws_secret_access_key`, or `aws_access_key_id`.
      * You must provide the model name in the `name` field.
      * When deploying Deploy on Demand (DoD) models, you need to explicitly provide the model configuration during registration. Set these configuration values according to the model’s requirements, since they don’t automatically transfer during inference from the watsonx Orchestrate side.
              <Expandable title="example">
                ```yaml YAML theme={null}
                config:
                    max_tokens: 2000
                    temperature: 0
                    decoding_method: "sample"
                ```
              </Expandable>
    </Note>
  </Accordion>

  <Accordion title="Mistral">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider.
        </ResponseField>

        <ResponseField name="mistral_fim_completion" type="boolean" />

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Define the model specification file">
        ```yaml mistral-large.yaml theme={null}
        spec_version: v1
        kind: model
        name: mistralai/mistral-7b-instruct-v0.3
        display_name: Mistral 7B Instruct v0.3
        description: |
            Mistral model for general-purpose reasoning and coding tasks.
        tags:
        - mistral
        model_type: chat
        provider_config:
            mistral_fim_completion: false
        ```
      </Step>

      <Step title="Create an API key connection">
        ```bash theme={null}
        orchestrate connections add -a mistral_credentials
        orchestrate connections configure -a mistral_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a mistral_credentials --env draft -e "api_key=my_mistral_api_key"
        ```
      </Step>

      <Step title="Add the model">
        ```bash theme={null}
        orchestrate models import --file mistral-large.yaml --app-id mistral_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="OpenRouter">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider.
        </ResponseField>

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Create an API key connection">
        ```bash theme={null}
        orchestrate connections add -a openrouter_credentials
        orchestrate connections configure -a openrouter_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a openrouter_credentials --env draft -e "api_key=my_openrouter_api_key"
        ```
      </Step>

      <Step title="Define the model specification file">
        ```yaml openrouter-model.yaml theme={null}
        spec_version: v1
        kind: model
        name: openrouter/openai/gpt-5
        display_name: OpenRouter GPT-5 Chat
        description: |
            OpenRouter model for routing requests across multiple LLM providers.
        tags:
        - openrouter
        - gpt
        model_type: chat
        provider_config: {}
        ```
      </Step>

      <Step title="Add the model">
        ```bash theme={null}
        orchestrate models import --file openrouter-model.yaml --app-id openrouter_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="x.ai">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider.
        </ResponseField>

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Create an API key connection">
        ```bash theme={null}
        orchestrate connections add -a xai_credentials
        orchestrate connections configure -a xai_credentials --env draft -k key_value -t team
        orchestrate connections set-credentials -a xai_credentials --env draft -e "api_key=xai_api_key"
        ```
      </Step>

      <Step title="Define the model specification file">
        ```yaml xai-model.yaml theme={null}
        spec_version: v1
        kind: model
        name: virtual-model/x-ai/grok
        display_name: Grok
        description: |
            x.ai model
        tags:
        - x.ai
        - gpt
        model_type: chat
        provider_config: {}
        ```
      </Step>

      <Step title="Add the model">
        ```bash theme={null}
        orchestrate models import --file xai-model.yaml --app-id xai_credentials
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="Ollama">
    <ResponseField name="provider_config" type="object">
      The fields which can either be set by connection or by the `provider_config` field of the model. Values from
      a connection will be merged with the `provider_config`.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="api_key" type="string" required="true">
          The API key for the provider
        </ResponseField>

        <ResponseField name="custom_host" type="string" required="true">
          Send requests to a custom hostname other than the default for the provider
        </ResponseField>

        <ResponseField name="url_to_fetch" type="string" post={["conditionally required"]}>
          The Ollama url to fetch the list of available ollama models
        </ResponseField>

        <ResponseField name="response_headers" type="list[string]">
          Add one or more additional response headers in the form `["header:value", "header2:value2"]` to the
          request to the server.
        </ResponseField>

        <ResponseField name="response_timeout" type="number">
          The response timeout in seconds
        </ResponseField>

        <ResponseField name="transform_to_form_data" type="boolean">
          Transforms the request to `form_data`.
        </ResponseField>
      </Expandable>
    </ResponseField>

    **Example usage:**

    <Steps>
      <Step title="Start ollama">
        In some systems, `ollama` might run under the `systemctl`, so you need to stop it before you run the Ollama server:

        ```
        systemctl stop ollama
        ```

        Then you can start the Ollama server and download the model, if it has not started yet:

        ```
        ollama pull llama3.2:latest
        export OLLAMA_HOST=0.0.0.0:11434
        ollama serve
        ```
      </Step>

      <Step title="Get your IP address">
        You can get your network IP address by running:

        <CodeGroup>
          ```powershell Windows theme={null}
          ipconfig # get the IPv4 address
          ```

          ```bash Linux theme={null}
          hostname -I # get the first address
          ```

          ```bash macOS theme={null}
          ipconfig getifaddr en0
          ```
        </CodeGroup>
      </Step>

      <Step title="Testing your connection">
        Before you import the model, it is a good idea to test your connection to guarantee the watsonx Orchestrate Developer Edition server can connect to the Ollama server.

        1. Use the following curl command to test your connection, replacing `198.51.100.42` with the IP address that you obtained in the previous step:

        ```bash theme={null}
        curl --request POST \
        --url http://198.51.100.42:11434/v1/chat/completions \
        --header 'content-type: application/json' \
        --data '{
        "model": "llama3.2:latest",
        "messages": [
        {
        "content": "Hi",
        "role": "user"
        }
        ]
        }'
        ```

        1. Enter the watsonx Orchestrate Developer Edition gateway container:

        ```bash theme={null}
        docker exec -it docker-wxo-agent-gateway-1 sh
        ```

        1. Run the curl command again from within the container shell.

        <Tip>
          **Tips:**
          If you experience connection issues with Ollama:

          * Wait a few minutes after starting the server before running the command.
          * Restart the Ollama server.
          * Close any VPN clients.
          * Try reconnecting to both Wi-Fi and wired Ethernet simultaneously.
          * Avoid switching networks during the process.
          * Reset the watsonx Orchestrate Developer Edition server:

          ```bash theme={null}
          orchestrate server reset
          ```
        </Tip>
      </Step>

      <Step title="Define the model specification file">
        For Ollama, you don't need to create a connection or use an actual API key. You can use a string such as `ollama` as an API key.

        You must use your current local network IP address as your URL. Ollama will not work if you use `localhost` or `0.0.0.0` in the model specification file.

        ```yaml ollama-llama2.yaml theme={null}
        spec_version: v1
        kind: model
        name: ollama/llama3.2:latest
        display_name: Ollama LLaMA 3.2
        description: |
            Ollama-hosted LLaMA 3.2 model for local or edge deployments.
        tags:
        - ollama
        - llama2
        model_type: chat
        provider_config:
            api_key: ollama
            custom_host: http://198.51.100.42:11434
        ```

        <Note>**Remember:** Replace `http://198.51.100.42:11434` with the IP address that you have obtained in the previous step.</Note>
      </Step>

      <Step title="Add the model">
        ```bash theme={null}
        orchestrate models import --file ollama-llama2.yaml
        ```
      </Step>
    </Steps>
  </Accordion>
</AccordionGroup>

### List all LLMs

Run the `orchestrate models list` command to see all available LLMs in your active environment.

```bash BASH theme={null}
orchestrate models list
```

<Note>
  **Note:**

  By default, you see a table of available models. If you prefer raw output, add the `--raw` (`-r`) flag.
</Note>

### Removing custom LLMs

Run the `orchestrate models remove` command and use the `--name` (`-n`) flag to specify the LLM you want to remove.

```bash BASH theme={null}
orchestrate models remove -n <model-name-unique-identifier-to-delete>
```

### Updating custom LLM

To update a custom LLM, first remove it, then add it again:

```bash BASH theme={null}
orchestrate models remove -n <model-name-unique-identifier-to-delete>
orchestrate models add --name groq/openai/gpt-oss-120b --app-id watsonx_ai_creds
```

### Additional configuration options

#### Setting a default LLM in the UI

If you use an on-premises installation with models provisioned only through the AI gateway, you can choose which model appears as the default in the user interface. To do this, add the `default` tag under the `tags` section of a model with the type set to `chat`.

```yaml granite-default-model.yaml [expandable] theme={null}
spec_version: v1
kind: model
name: watsonx/ibm/granite-3.3-8b-instruct
display_name: IBM watsonx.ai (Granite)
description: |
    IBM watsonx.ai model using Space-scoped configuration.
tags:
- default # <-- this marks this as the Default model in the ui dropdown
model_type: chat
provider_config:
    watsonx_space_id: my-space-id
```

<Note>
  **Note:**

  For on-premises installations using only externally hosted virtual-models, at least one model must be specified as the default model or it will not be possible
  to open the "Create Agent" page in the UI.
</Note>

#### Setting a default LLM for knowledge bases

If you use an on-premises installation with models provisioned only through the AI gateway, you can also set a default model for knowledge bases. To do this, add the `default` tag under the `tags` section of a model with the type set to `embedding`.

```yaml virtual-model.yaml [expandable] theme={null}
spec_version: v1
kind: model
name: virtual-model/watsonx/ibm/slate-30m-english-rtrvr-v2
display_name: slate30m
tags:
  - default
model_type: embedding
provider_config:
  watsonx_space_id: xxx
  customHost: 'https://us-south.ml.cloud.ibm.com'
  api_key: xxx
```