Exporting observability traces with Observability SDK

This topic describes how to use the ibm-watsonx-orchestrate-sdk observability decorators to instrument a LangGraph-based agent. LangGraph is a Python framework for building stateful, graph-based AI agents whose logic is expressed as nodes in a StateGraph. The SDK manages span creation, nesting, authentication, and export internally, so only a few decorator additions to your agent code are required. Use this approach when:

Your agent is built with LangGraph.
You want the finest level of trace detail: per-LLM-call, per-tool-call, and per-agent-node spans with structured metadata.
You prefer decorator-based instrumentation over manually writing tracer and exporter code.

For an approach that requires no SDK dependency and uses the standard OpenTelemetry SDK instead, see Exporting observability traces with OpenTelemetry.

How it works

The SDK wraps the same OpenTelemetry export pipeline as the OpenTelemetry export approach, but manages it for you:

Your LangGraph agent
│
├─ @configure_tracing       ← wires tracing context into the compiled graph
├─ @trace_agent_call        ← root span per agent node invocation
│   ├─ @trace_llm_call      ← child span per LLM invocation
│   └─ @trace_tool_call     ← child span per tool invocation
│       └─ @trace_call      ← child span for any helper function
│
└─ SDK (internal)
    ├─ Client  (authenticates using api_key + instance_url)
    ├─ TracerProvider + DynamicAuthOTLPSpanExporter
    └─ ──OTLP/HTTP──►  watsonx Orchestrate ingestion endpoint
                        └─► Analytics dashboard
                            Overview / Conversations

The setup follows four steps:

Initialize a Client with your API key and instance URL.
Create a TracerConfig that identifies the agent and workspace that the exported spans belong to, by using the registered agent’s agent_id, workspace_id, and environment. On AWS deployments, the SDK derives the tenant context automatically from the JWT token. On IBM Cloud, pass tenant_id explicitly because the IAM token does not contain tenant information.
Build a Tracer from the configuration and register it globally with register_tracer() to make it available to all SDK decorators.
Apply decorators to the relevant functions. Each decorator is a Python @ annotation that intercepts function calls at runtime to create and export spans without modifying your function logic.

After watsonx Orchestrate receives the spans, they appear in Analyze > [Agent name] > Overview and Conversations, showing total conversations, token counts, average duration, and the usage trend chart.

Before you begin

Complete the following setup before you instrument your agent:

Import the LangGraph agent through the watsonx Orchestrate UI to create an agent entry and obtain a valid Agent ID. For more information, see Importing LangGraph agents.
Copy the Agent ID from the registered agent.
For IBM Cloud: Copy the Workspace ID. Call the List workspaces API to fetch the workspaces from a tenant.
Copy the Tenant ID for your instance. The Tenant ID has the format <account-id>_<instance-id>. Obtain it in one of the following ways:
- From the instance CRN in Profile > About.
- From the browser DevTools console, run WO_Meta to retrieve its properties.
When you use the SDK on AWS deployments, the tenant context is derived automatically from the JWT token inside Client. On IBM Cloud, the IAM token does not contain tenant information, so you must pass the Tenant ID explicitly as tenant_id in TracerConfig.
Obtain an API key and the corresponding IAM or MCSP token URL that is authorized to call the trace ingestion endpoint. For more information, see Authenticating to the API.
Note the instance URL for your watsonx Orchestrate instance. This URL is the base URL of the ingestion endpoint and has the format https://api.watson-orchestrate.ibm.com/instances/<instance-id>. For more information, see Getting the API endpoint.

Save the Agent ID, Tenant ID, API key, token URL, and instance URL values for use in the procedure.

Procedure

Installing the SDK

Because the SDK is currently a pre-release build, install it from Test PyPI.To perform a quick install, run:

BASH

pip install -i https://test.pypi.org/simple/ ibm-watsonx-orchestrate-sdk

For a project with a requirements.txt, add the following entries:

# requirements.txt

# Resolve all other packages from public PyPI
--extra-index-url https://pypi.org/simple
--index-url https://test.pypi.org/simple/

# Add your agent's dependencies here, for example:
requests>=2.31.0

# WXO Observability SDK (pre-release, Test PyPI)
ibm-watsonx-orchestrate-sdk @ https://test-files.pythonhosted.org/packages/70/8a/04bfbb0e81c369023cbb8a562d81d69c9df27b776b6488a86b55da8d1545/ibm_watsonx_orchestrate_sdk-2.14.0.dev7756-py3-none-any.whl

Then install the dependencies:

BASH

pip install -r requirements.txt

Configuring the tracer

At module load time, create a Client, build a TracerConfig, and register the tracer globally. Complete this step once, before any decorated functions are called.

AWS
IBM Cloud

Python

from ibm_watsonx_orchestrate_sdk.client import Client
from ibm_watsonx_orchestrate_sdk.observability import (
    Tracer,
    TracerConfig,
    register_tracer,
)

# Authenticate and identify the WXO instance
client = Client(
    api_key="<your-api-key>",
    instance_url="https://api.watson-orchestrate.ibm.com/instances/<instance-id>",
)

# Identify the agent and workspace this tracer belongs to
# On AWS, tenant_id is extracted automatically from the JWT token inside client
config = TracerConfig(
    client=client,
    agent_id="<your-agent-id>",       # Agent ID from registration step
    workspace_id="<your-workspace-id>",
    environment="live",               # "draft" or "live"
)

tracer = Tracer(config)
register_tracer(tracer)

Python

from ibm_watsonx_orchestrate_sdk.client import Client
from ibm_watsonx_orchestrate_sdk.observability import (
    Tracer,
    TracerConfig,
    register_tracer,
)

# Authenticate and identify the WXO instance
client = Client(
    api_key="<your-api-key>",
    instance_url="https://api.watson-orchestrate.ibm.com/instances/<instance-id>",
)

# On IBM Cloud, the IAM token does not contain tenant information,
# so tenant_id must be provided explicitly in TracerConfig.
config = TracerConfig(
    client=client,
    tenant_id="<your-tenant-id>",     # Required on IBM Cloud; format: <account-id>_<instance-id>
    agent_id="<your-agent-id>",       # Agent ID from registration step
    workspace_id="<your-workspace-id>",
    environment="live",               # "draft" or "live"
)

tracer = Tracer(config)
register_tracer(tracer)

Key points:

Pass api_key and instance_url to Client.
On AWS, the SDK derives the tenant context automatically from the JWT token, so tenant_id is not required.
On IBM Cloud, the IAM API key generates a user-level token that does not include tenant information. Pass tenant_id explicitly in TracerConfig to avoid a ValueError: Trace injection mode requires the following parameters: tenant_id at startup.
The only valid values for environment are "draft" and "live".
Tracer uses DynamicAuthOTLPSpanExporter to refresh tokens automatically. Long-running agent processes do not need to be restarted when an access token expires.

Decorating agent functions

Apply the SDK decorators to the functions that make up your agent. Manual span creation is not required.The following table shows which decorator to apply and where:

Decorator	Apply to	What it captures
`@configure_tracing`	The factory function that builds the `StateGraph`	Wires tracing context through the compiled graph
`@trace_agent_call`	The top-level LangGraph node function	Root span for the agent decision step
`@trace_llm_call`	The function that calls the LLM	LLM invocation with model, provider, and token counts
`@trace_tool_call`	Functions also decorated with `@tool`	Tool invocations with tool-specific metadata
`@trace_call`	Any other helper function	Input, output, and custom attributes

General-purpose helper calls: `@trace_call`

Use @trace_call for helper logic such as validation or response formatting. Set capture_input=True or capture_output=True to record function arguments and return values as span attributes.

Python

from ibm_watsonx_orchestrate_sdk.observability.decorators import trace_call

@trace_call(
    name="validate_currency",
    capture_input=True,
    capture_output=True,
    attributes={"validation_type": "currency_support"},
)
def validate_currency(currency_code: str) -> tuple[bool, str]:
    ...

Tool calls: `@trace_tool_call`

Use @trace_tool_call on functions that are also decorated with LangChain’s @tool decorator. The SDK records each tool invocation as a distinct span with tool-specific metadata.

@tool must be the outermost decorator. Place @tool above @trace_tool_call in the source file, with @trace_tool_call applied directly to the function.

Python

from langchain_core.tools import tool
from ibm_watsonx_orchestrate_sdk.observability.decorators import trace_tool_call

@tool
@trace_tool_call(
    name="get_exchange_rate_tool",
    capture_input=True,
    capture_output=True,
    tool_name="get_exchange_rate",
    attributes={"api": "frankfurter", "version": "v2"},
)
def get_exchange_rate(base_currency: str, quote_currency: str) -> str:
    ...

LLM calls: `@trace_llm_call`

Wrap the function that invokes the LLM with @trace_llm_call. Set model and provider so that the analytics view correctly attributes token and latency metrics.

Python

from ibm_watsonx_orchestrate_sdk.observability.decorators import trace_llm_call

@trace_llm_call(
    name="llm_call",
    capture_input=True,
    capture_output=True,
    model="openai/gpt-4o-mini",
    provider="openai",
    attributes={"agent_type": "currency_exchange"},
)
def _invoke_llm(llm, messages):
    """Invoke the LLM with tracing."""
    return llm.invoke(messages)

Agent node: `@trace_agent_call`

Apply @trace_agent_call to the top-level LangGraph node that represents the agent decision step. The agent_name parameter labels this trace in the analytics view. The agent_id in TracerConfig identifies the watsonx Orchestrate agent instance to which the trace belongs.

Python

from ibm_watsonx_orchestrate_sdk.observability.decorators import trace_agent_call
from langgraph.graph import StateGraph
from langchain_core.runnables import RunnableConfig

@trace_agent_call(
    name="agent_node",
    agent_name="my-external-agent",
    framework="langgraph",
    capture_input=True,
    capture_output=True,
    attributes={"agent_type": "currency_exchange", "has_tools": "true"},
)
def agent_node(state: AgentState, config: RunnableConfig) -> AgentState:
    ...

Graph factory: `@configure_tracing`

Apply @configure_tracing to the factory function that builds and returns the LangGraph StateGraph. This decorator propagates tracing context through the compiled graph.

Python

from ibm_watsonx_orchestrate_sdk.observability.decorators import configure_tracing

@configure_tracing
def create_agent(config: RunnableConfig) -> StateGraph:
    workflow = StateGraph(AgentState)
    workflow.add_node("agent", agent_node_with_config)
    workflow.add_node("tools", ToolNode([get_exchange_rate]))
    workflow.add_edge(START, "agent")
    workflow.add_conditional_edges(
        "agent", should_continue, {"tools": "tools", "end": END}
    )
    workflow.add_edge("tools", "agent")
    return workflow

Result

After the decorated agent runs, the SDK exports the captured spans to watsonx Orchestrate by using the Client credentials and the configured agent_id and tenant context. The traces are visible in watsonx Orchestrate under Analyze > [Agent name] > Overview and Conversations. Per-call span detail is available in the Conversations drill-down view.

Troubleshooting

Verifying trace ingestion

If traces do not appear on the Analytics page, use the Get Traces API to confirm whether the ingestion endpoint received them. If the API returns your traces, ingestion was successful. If the Get Traces API returns an empty list, the spans were not received. Check the following items:

401 Unauthorized on startup: The API key or instance URL passed to Client is incorrect. Verify that both values match the watsonx Orchestrate instance where the agent is registered.
Token expiry during a long run: The DynamicAuthOTLPSpanExporter refreshes tokens automatically. If authentication failures persist, confirm that the API key has not been revoked and that the instance URL is reachable from the agent host.
agent_id or workspace_id not found: The values passed to TracerConfig must match the Agent ID and Workspace ID shown in the watsonx Orchestrate UI for the registered agent. Verify both values against the registration entry.
@configure_tracing not applied: If @configure_tracing is missing from the graph factory function, tracing context is not propagated through the compiled graph, and child spans may not be associated with the correct root span.

Release Notes

Get Started

Build

Deploy

Analyze

Developer experience

Legal notices

Exporting observability traces with Observability SDK

How it works

Before you begin

Procedure

Installing the SDK

Configuring the tracer

Decorating agent functions

General-purpose helper calls: `@trace_call`

Tool calls: `@trace_tool_call`

LLM calls: `@trace_llm_call`

Agent node: `@trace_agent_call`

Graph factory: `@configure_tracing`

Result

Troubleshooting

Verifying trace ingestion

​How it works

​Before you begin

​Procedure

Installing the SDK

Configuring the tracer

Decorating agent functions

​General-purpose helper calls: @trace_call

​Tool calls: @trace_tool_call

​LLM calls: @trace_llm_call

​Agent node: @trace_agent_call

​Graph factory: @configure_tracing

​Result

​Troubleshooting

Verifying trace ingestion

How it works

Before you begin

Procedure

General-purpose helper calls: `@trace_call`

Tool calls: `@trace_tool_call`

LLM calls: `@trace_llm_call`

Agent node: `@trace_agent_call`

Graph factory: `@configure_tracing`

Result

Troubleshooting