> ## Documentation Index > Fetch the complete documentation index at: https://developer.watson-orchestrate.ibm.com/llms.txt > Use this file to discover all available pages before exploring further. # Exporting observability traces with OpenTelemetry > Surface execution traces from agents running outside watsonx Orchestrate in the Agent Analytics dashboard by using any agentic framework, without requiring a watsonx Orchestrate SDK dependency. This topic describes how to instrument your agent by using the standard OpenTelemetry Python SDK and export spans over [OTLP/HTTP](https://opentelemetry.io/docs/specs/otlp/) to the watsonx Orchestrate trace ingestion endpoint. Use this approach when: * Your agent is built with any agentic framework. * You prefer not to add a watsonx Orchestrate-specific SDK dependency. * You want a high-level view of requests and responses, without per-call detail such as LLM calls or tool calls. For an approach that uses the `ibm-watsonx-orchestrate-sdk` observability decorators to instrument a LangGraph-based agent, see [Exporting observability traces with the Observability SDK](observability-sdk). ## How it works The following diagram shows the data flow from your agent to the Analytics dashboard: ``` Your agent process │ ├─ TracerProvider (configured once at startup) │ └─ BatchSpanProcessor │ └─ OTLPSpanExporter ──OTLP/HTTP──► watsonx Orchestrate │ trace ingestion endpoint │ │ └─ tracer.start_as_current_span(...) └─► Analytics dashboard creates spans during the run Overview / Conversations ``` The setup follows four steps: 1. Initialize a `TracerProvider` with a `Resource` that describes your service. 2. Configure an `OTLPSpanExporter` with the watsonx Orchestrate trace ingestion URL and an `Authorization` header containing a bearer token obtained from the IAM or MCSP token endpoint. 3. Create spans manually by using `tracer.start_as_current_span(...)`. Nest spans to reflect the logical steps of the agent run. The outermost span is the root span and represents the entire agent run. Child spans represent individual steps. 4. Attach a `BatchSpanProcessor` and call `force_flush()` at the end of the run to drain the buffer before the process exits. After watsonx Orchestrate receives the spans, they appear in **Analyze** > **\[Agent name]** > **Overview** and **Conversations**, showing total conversations, token counts, average duration, and the usage trend chart. ## Prerequisites Complete the following setup before you instrument your agent: 1. Register your agent in watsonx Orchestrate to create an agent entry and obtain a valid Agent ID. Use one of the following CLI methods: * **YAML import**: For any agent type, create an agent configuration file with `kind: external` and run: ```bash BASH theme={null} orchestrate agents import -f ``` For details on authoring the configuration file, see [Connect to external agents](../agents/connect_agent). * **Agent discovery**: For A2A-compatible agents that publish an agent card, run: ```bash BASH theme={null} orchestrate agents discover -u ``` For details, see [Import external agents from a URL](../agents/connect_agent#import-external-agents-from-a-url). For a full reference of import and create options, see [Importing and deploying agents](../agents/import_agent). 2. Copy the **Agent ID** from the registered agent. 3. Copy the **Tenant ID** for your instance. The Tenant ID has the format `_`. Obtain it in one of the following ways: * From the instance CRN in **Profile** > **About**. * From the browser DevTools console, run `WO_Meta` to retrieve its properties. 4. Obtain an **API key** and the corresponding **IAM or MCSP token URL** that is authorized to call the trace ingestion endpoint. For more information, see [Authenticating to the API](https://www.ibm.com/docs/en/watsonx/watson-orchestrate/base?topic=api-getting-started#api-key-getting-started__authenticating-to-the-api__title__1). 5. Note the **instance URL** for your watsonx Orchestrate instance. This URL is the base URL of the ingestion endpoint and has the format `https://api.watson-orchestrate.ibm.com/instances/`. For more information, see [Getting the API endpoint](https://www.ibm.com/docs/en/watsonx/watson-orchestrate/base?topic=api-getting-endpoint). Save the Agent ID, Tenant ID, API key, token URL, and instance URL values for use in the procedure. ## Procedure Install the standard OpenTelemetry packages listed in the following table. No watsonx Orchestrate-specific package is required. ```bash BASH theme={null} pip install \ opentelemetry-api \ opentelemetry-sdk \ opentelemetry-exporter-otlp-proto-http ``` | Package | Purpose | | ---------------------------------------- | ---------------------------------------------------------------- | | `opentelemetry-api` | Provides the `trace` API and span interfaces. | | `opentelemetry-sdk` | Provides `TracerProvider`, `Resource`, and `BatchSpanProcessor`. | | `opentelemetry-exporter-otlp-proto-http` | Provides `OTLPSpanExporter`, which sends spans over OTLP/HTTP. | Set the following environment variables in the process that runs your external agent. You can define them in a shell script, a Kubernetes Secret, or a CI/CD variable store. | Variable | Description | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `TENANT_ID` | The watsonx Orchestrate tenant identifier in the format `_`. Obtain it from the instance CRN or by running `WO_Meta` in the browser console. | | `AGENT_ID` | The ID of the agent registered in the watsonx Orchestrate UI. | | `ENVIRONMENT_NAME` | The deployment environment label, for example `draft` or `live`. | | `OTEL_RESOURCE_ATTRIBUTES` | Comma-separated resource attributes attached to every span. Must include `tenant.id` and `deployment.environment` so that the ingestion endpoint can route traces to the correct tenant. Set to `tenant.id=$TENANT_ID,deployment.environment=$ENVIRONMENT_NAME`. | | `API_KEY` | The API key used to obtain a bearer token for the ingestion endpoint. | | `TOKEN_URL` | The IAM or MCSP endpoint used to exchange `API_KEY` for a bearer token. | | `OTEL_EXPORT_URL` | The OTLP/HTTP trace ingestion endpoint for the target watsonx Orchestrate instance. | The following example shell script sets all required variables. Source it before you start the agent process. ```bash BASH wrap theme={null} #!/usr/bin/env bash set -euo pipefail cd "$(dirname "$0")" source .venv/bin/activate export TENANT_ID="" export AGENT_ID="" export ENVIRONMENT_NAME="draft" export OTEL_RESOURCE_ATTRIBUTES="tenant.id=${TENANT_ID},deployment.environment=${ENVIRONMENT_NAME}" export API_KEY="" export TOKEN_URL="https://iam.platform.saas.ibm.com/siusermgr/api/1.0/apikeys/token" export OTEL_EXPORT_URL="https://api.watson-orchestrate.ibm.com/instances//v1/orchestrate/inject/traces" ``` Treat `API_KEY` as a secret. Inject it by using a secrets manager or CI/CD variable store rather than committing it to source control. `OTEL_RESOURCE_ATTRIBUTES` is a standard OpenTelemetry environment variable that the SDK reads automatically. It adds `tenant.id` and `deployment.environment` to every span so that the ingestion endpoint routes the trace to the correct tenant. The code in the next step also calls `Resource.create(RESOURCE_ATTRIBUTES)` with a Python dictionary that sets `service.name`, `service.version`, and `application`. The SDK merges both sets of attributes so that all of them appear on every span. Define the service identity and the per-run attributes that watsonx Orchestrate requires to associate each span with the correct agent and conversation. The SDK merges `RESOURCE_ATTRIBUTES` with the attributes defined in `OTEL_RESOURCE_ATTRIBUTES` in the previous step, so both the service identity and tenant routing attributes appear on every span. ```python Python wrap expandable theme={null} import uuid import os RESOURCE_ATTRIBUTES = { "service.name": "my-external-agent", "service.version": "1.0.0", "application": "my-agent-server", } def get_required_env(name: str) -> str: value = os.environ.get(name) if not value: raise RuntimeError(f"Missing required environment variable: {name}") return value def build_span_attributes() -> dict[str, str]: return { "agent.id": get_required_env("AGENT_ID"), "langfuse.user.id": f"usr_{uuid.uuid4().hex[:12]}", "langfuse.session.id": str(uuid.uuid4()), } ``` Set the attributes that are returned by `build_span_attributes()` on the root span of every agent run: | Attribute | Value | Description | | --------------------- | --------------------------- | --------------------------------------------------------------------------------- | | `agent.id` | String | The Agent ID from the watsonx Orchestrate UI. | | `langfuse.session.id` | UUID string | Groups all spans in a single conversation turn. Generate a new UUID for each run. | | `langfuse.user.id` | String prefixed with `usr_` | Identifies the end user of the conversation turn. | Exchange your API key for a bearer token, and then construct the `OTLPSpanExporter` and `TracerProvider`. ```python Python wrap expandable theme={null} import json import ssl from urllib.request import Request, urlopen from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor def get_bearer_token() -> str: """Exchange API_KEY for a bearer token via the IAM/MCSP token endpoint.""" api_key = get_required_env("API_KEY") token_url = get_required_env("TOKEN_URL") payload = json.dumps({"apikey": api_key}).encode("utf-8") request = Request( token_url, data=payload, headers={ "Content-Type": "application/json", "Accept": "application/json", }, method="POST", ) with urlopen(request, timeout=30, context=ssl.create_default_context()) as response: token_response = json.loads(response.read().decode("utf-8")) bearer_token = token_response.get("token") or token_response.get("access_token") if not bearer_token: raise RuntimeError(f"Token response missing token/access_token: {token_response}") return str(bearer_token) def setup_tracer() -> tuple[trace.Tracer, TracerProvider]: otel_export_url = get_required_env("OTEL_EXPORT_URL") resource = Resource.create(RESOURCE_ATTRIBUTES) provider = TracerProvider(resource=resource) bearer_token = get_bearer_token() exporter = OTLPSpanExporter( endpoint=otel_export_url, headers={"Authorization": f"Bearer {bearer_token}"}, ) processor = BatchSpanProcessor(exporter) provider.add_span_processor(processor) trace.set_tracer_provider(provider) return trace.get_tracer("wxo-server"), provider ``` ##### Token lifetime IAM and MCSP bearer tokens are short-lived. `get_bearer_token()` fetches the token once at startup. This is sufficient for short-lived scripts or request handlers that complete within the token lifetime. When a token expires, the ingestion endpoint returns `401 Unauthorized`. For long-running processes, catch this response, re-fetch the token, rebuild the exporter with the new token, and retry the request. The following authentication paths are also supported: * **`MCSP_v2`**: Uses the `X_API_KEY` and `MCSP_V2_TOKEN_URL` environment variables for SaaS environments that require the MCSP v2 token type. * **Legacy `x-api-key`**: Passes the API key directly in an `x-api-key` request header without exchanging it for a bearer token first. Use this path only if your instance does not support token-based authentication. Wrap each logical step of your agent's execution in a span, and nest child spans to reflect the call hierarchy. Call `build_span_attributes()` to retrieve the root span attributes, and then set them on the root span before you create any child spans. ```python Python wrap expandable theme={null} from typing import cast def main() -> None: tracer, provider = setup_tracer() span_attributes = build_span_attributes() with tracer.start_as_current_span("sample-operation") as span: for key, value in span_attributes.items(): span.set_attribute(key, value) span.set_attribute("input", "User asked Agent to greet and start an orchestrate run") span.set_attribute("output", "Agent prepared a greeting response and submitted the orchestrate run") # Child span: load conversation context with tracer.start_as_current_span("load-conversation-context") as child_span: child_span.set_attribute("operation.type", "context_lookup") # Child span: resolve and call the agent with tracer.start_as_current_span("resolve-agent") as child_span: child_span.set_attribute("operation.type", "agent_resolution") with tracer.start_as_current_span("prepare-agent-response") as nested_span: nested_span.set_attribute("operation.type", "response_preparation") with tracer.start_as_current_span("calling-agent") as nested_span: nested_span.set_attribute("operation.type", "response_preparation") # Child span: submit the orchestrate run with tracer.start_as_current_span("submit-orchestrate-run") as child_span: child_span.set_attribute("operation.type", "http_request") child_span.set_attribute("http.method", "POST") child_span.set_attribute("http.route", "/orchestrate/runs") # Flush all buffered spans before the process exits cast(TracerProvider, provider).force_flush() if __name__ == "__main__": main() ``` Always call `force_flush()` at the end of the run. If you omit this call, the `BatchSpanProcessor` background thread might not flush all pending spans before the process exits, and trace data may be lost. ## Result After the script completes and flushes successfully, the conversation and its nested spans are visible in watsonx Orchestrate under **Analyze** > **\[Agent name]** > **Overview** and **Conversations**. ## Troubleshooting ##### Verifying trace ingestion If traces do not appear on the Analytics page, use the [Get Traces API](https://developer.ibm.com/apis/catalog/watsonorchestrate--custom-assistants/api/API--watsonorchestrate--agentops-and-analytics#getTraces) to confirm whether the ingestion endpoint received them. If the API returns your traces, ingestion was successful. If the Get Traces API returns an empty list, the spans were not received. Check the following items: * **`401 Unauthorized`**: The bearer token has expired. Fetch a new token and rebuild the exporter as described in [Authenticating and configuring the exporter](#authenticating-and-configuring-the-exporter). * **Spans missing after process exit**: `force_flush()` was not called. The `BatchSpanProcessor` background thread may not have flushed all pending spans before the process exited. Ensure `force_flush()` is called at the end of every run. * **Wrong endpoint**: Confirm that `OTEL_EXPORT_URL` points to the correct instance URL. The correct format is `https://api.watson-orchestrate.ibm.com/instances//v1/orchestrate/inject/traces`. * **Missing resource attributes**: Confirm that `OTEL_RESOURCE_ATTRIBUTES` includes both `tenant.id` and `deployment.environment`. Without these attributes, the ingestion endpoint cannot route the trace to the correct tenant.