Connecting a knowledge base to an external content source (Public Preview)

Important:This feature is currently in public preview. Functionality and behavior may change in future updates.

Use a connector-backed knowledge base to ingest documents from a third-party content source such as Box, SharePoint, or OneDrive. You create a connection for the source credentials, identify source paths in the knowledge base specification, and import the specification with the ADK CLI. watsonx Orchestrate supports only key_value connections. Use --kind key_value or --kind kv for each source. Other OAuth connection kinds are not validated end-to-end for knowledge base content sources. A connector-backed knowledge base does not upload document files through the ADK. The knowledge base specification identifies the external system in content_source and identifies the source paths in documents. During a sync, watsonx Orchestrate retrieves the files, converts and chunks their content, and indexes the chunks in the built-in vector store. The source documents must meet these requirements:

Give each file a unique name.
Include no more than 100 files in one knowledge base specification.
Limit each .xlsx file to 1 MB.
Limit each .docx, .pdf, and .pptx file to 25 MB.
Limit each .csv, .html, and .txt file to 5 MB.

Before you begin

Register an application or service account with your content provider.
Collect the required credentials from the provider.
If you use Developer Edition, start the server with connector support:

BASH

orchestrate server start -e <path-to-env-file> --with-ingestion-from-external-source

Required credentials

Each third-party source requires a different set of credentials from its own developer console or app registration. Pass each credential as a separate --entries "<key>=<value>" option on orchestrate connections set-credentials.

content_source.type	Required key/value fields
`box`	`client_id`, `client_secret`, `enterprise_id`, `public_key`, `private_key`, `private_key_password`
`sharepoint-file`	`host`, `site`, `tenant_id`, `client_id`, `client_secret`, `refresh_token`
`onedrive`	`tenant_id`, `client_id`, `client_secret`, `username`, `password`

Connecting to a third-party content source

Create a connection

Create a connection with an application ID that identifies the external source:

BASH

orchestrate connections add -a <app-id>

Configure a team connection in the draft environment:

BASH

orchestrate connections configure \
  -a <app-id> \
  --kind key_value \
  --type team \
  --env draft

Set the credentials. Use the fields for your source from the required credentials table:

BASH

orchestrate connections set-credentials \
  -a <app-id> \
  --entries "<field1>=<value1>" \
  --entries "<field2>=<value2>" \
  --env draft

To use the knowledge base in the live environment, configure the connection and set its credentials again with --env live.

Note:Keep private keys and passwords out of source control. The connection stores these values separately from the knowledge base specification.

Define the knowledge base

Create a YAML file with content_source and documents:

YAML

spec_version: v1
kind: knowledge_base
name: <knowledge-base-name>
description: <description>
content_source:
  type: <source-type>
documents:
  - <path/to/file.ext>
  - <path/to/folder/>

string

required

The connector to use. Accepted values are box, sharepoint-file, and onedrive.

array

required

List of file and folder paths in the source system. Add a trailing / to a path to identify a folder — the sync ingests files directly in that folder but not files in its subfolders.

Do not set content_source.connection_id in the specification. The import command resolves the connection ID automatically from the connection that matches --app-id.

Import and perform the initial sync

Import the specification and provide the application ID from the connection:

BASH

orchestrate knowledge-bases import \
  -f <path-to-knowledge-base.yaml> \
  -a <app-id>

On the first import, watsonx Orchestrate starts a sync and the CLI waits for it to finish.When you import an existing knowledge base, the command updates the specification without starting a sync. Add --sync to update the specification and sync its content:

BASH

orchestrate knowledge-bases import \
  -f <path-to-knowledge-base.yaml> \
  -a <app-id> \
  --sync

If the sync fails, the knowledge base remains available. Correct the credentials or connector configuration, and then start another sync.

Check status and sync content

Check the detailed status:

BASH

orchestrate knowledge-bases status \
  --name <knowledge-base-name> \
  --verbose

The response includes sync_state, sync_state_msg, and last_sync_stats for a connector-backed knowledge base.

`sync_state`	Meaning
`started`	The platform accepted the sync request and started retrieving files.
`in_progress`	The platform processes document changes and builds embeddings.
`ready_for_promotion`	The platform prepared the new index for promotion.
`stable`	No sync is active and the production index is healthy.
`failed`	The latest sync failed. Check `sync_state_msg`, correct the problem, and sync again.

A detailed response can contain these fields:

JSON

{
  "id": "<knowledge-base-id>",
  "name": "<knowledge-base-name>",
  "ready": true,
  "built_in_index_status": "ready",
  "sync_state": "stable",
  "sync_state_msg": "Last sync completed. 1 added (/path/to/file.ext).",
  "last_sync_stats": {
    "added": ["/path/to/file.ext"],
    "deleted": [],
    "modified": [],
    "not_found": []
  },
  "documents": [
    {
      "id": "<document-id>",
      "metadata": {
        "source_uri": "<source-type>:///path/to/file.ext",
        "source_type": "<source-type>"
      }
    }
  ]
}

After a sync, use the verbose status output to confirm which files were indexed. If a path does not exist in the source, the import still succeeds but watsonx Orchestrate does not index those files. Correct the paths in the specification and re-import with --sync.Start an on-demand sync after you change files in the source or update the document paths:

BASH

orchestrate knowledge-bases sync --name <knowledge-base-name>

The command waits for the sync to finish. A knowledge base supports one active sync at a time. During a sync, the knowledge base continues to answer questions with the previous index. A successful sync promotes the new index. A failed sync keeps the previous index available.

Remove the knowledge base and connection

Remove the knowledge base:

BASH

orchestrate knowledge-bases remove --name <knowledge-base-name>

This command removes the indexed content from watsonx Orchestrate. It does not remove files from the external source. Remove the connection when you no longer need it:

BASH

orchestrate connections remove -a <app-id>

Examples

Box
SharePoint
OneDrive

Box uses a JWT service account. Get the credential values from your Box Developer Console application and its generated key pair.

BASH

orchestrate connections add -a box

orchestrate connections configure -a box --kind key_value --type team --env draft

orchestrate connections set-credentials -a box \
  --entries "client_id=<box_client_id>" \
  --entries "client_secret=<box_client_secret>" \
  --entries "enterprise_id=<box_enterprise_id>" \
  --entries "public_key=<box_public_key_id>" \
  --entries "private_key=<box_private_key_pem>" \
  --entries "private_key_password=<box_private_key_passphrase>" \
  --env draft

Create the knowledge base specification:

YAML

spec_version: v1
kind: knowledge_base
name: <kb-name>
description: <description>
content_source:
  type: box
documents:
  - <path/to/file.ext>
  - <path/to/folder/>

Import the specification:

BASH

orchestrate knowledge-bases import -f <path/to/knowledge_base.yaml> -a box

SharePoint uses a Microsoft Entra ID application registration and a delegated OAuth refresh token that you store as a key/value credential.

BASH

orchestrate connections add -a sharepoint

orchestrate connections configure -a sharepoint --kind key_value --type team --env draft

orchestrate connections set-credentials -a sharepoint \
  --entries "host=<yourtenant.sharepoint.com>" \
  --entries "site=<site-name>" \
  --entries "tenant_id=<azure_tenant_id>" \
  --entries "client_id=<azure_app_client_id>" \
  --entries "client_secret=<azure_app_client_secret>" \
  --entries "refresh_token=<oauth_refresh_token>" \
  --env draft

Create the knowledge base specification:

YAML

spec_version: v1
kind: knowledge_base
name: <kb-name>
description: <description>
content_source:
  type: sharepoint-file
documents:
  - <path/to/file.ext>
  - <path/to/folder/>

Import the specification:

BASH

orchestrate knowledge-bases import -f <path/to/knowledge_base.yaml> -a sharepoint

OneDrive uses a Microsoft Entra ID application registration with a username and password.

BASH

orchestrate connections add -a onedrive

orchestrate connections configure -a onedrive --kind key_value --type team --env draft

orchestrate connections set-credentials -a onedrive \
  --entries "tenant_id=<azure_tenant_id>" \
  --entries "client_id=<azure_app_client_id>" \
  --entries "client_secret=<azure_app_client_secret>" \
  --entries "username=<user@yourdomain.onmicrosoft.com>" \
  --entries "password=<user_password>" \
  --env draft

Create the knowledge base specification:

YAML

spec_version: v1
kind: knowledge_base
name: <kb-name>
description: <description>
content_source:
  type: onedrive
documents:
  - <path/to/file.ext>
  - <path/to/folder/>

Import the specification:

BASH

orchestrate knowledge-bases import -f <path/to/knowledge_base.yaml> -a onedrive

Troubleshooting

Problem	Resolution
The connection application ID already exists.	Use the existing connection or choose another application ID.
The CLI reports `No connection exists with the app-id`.	Create the connection or correct the value passed to `--app-id`.
The CLI reports that sync is unavailable in Developer Edition.	Restart Developer Edition with `--with-ingestion-from-external-source`.
`last_sync_stats.not_found` contains paths.	Correct the source paths in `documents`, import the specification again, and add `--sync`.
`sync_state` is `failed`.	Review `sync_state_msg`, correct the credentials or connector configuration, and run `orchestrate knowledge-bases sync`.
The CLI reports that the knowledge base is not connector-backed.	Use `knowledge-bases sync` only with a knowledge base that defines `content_source`.

Release Notes

Get Started

Build

Deploy

Analyze

Developer experience

Legal notices

Connecting a knowledge base to an external content source (Public Preview)

Before you begin

Required credentials

Connecting to a third-party content source

Remove the knowledge base and connection

Examples

Troubleshooting

​Before you begin

​Required credentials

​Connecting to a third-party content source

​Remove the knowledge base and connection

​Examples

​Troubleshooting

Before you begin

Required credentials

Connecting to a third-party content source

Remove the knowledge base and connection

Examples

Troubleshooting