> ## Documentation Index > Fetch the complete documentation index at: https://developer.watson-orchestrate.ibm.com/llms.txt > Use this file to discover all available pages before exploring further. # Exporting observability traces with Observability SDK > Surface execution traces from LangGraph agents running outside watsonx Orchestrate in the Agent Analytics dashboard by using decorator-based instrumentation. This topic describes how to use the `ibm-watsonx-orchestrate-sdk` observability decorators to instrument a LangGraph-based agent. [LangGraph](https://langchain-ai.github.io/langgraph/) is a Python framework for building stateful, graph-based AI agents whose logic is expressed as nodes in a `StateGraph`. The SDK manages span creation, nesting, authentication, and export internally, so only a few decorator additions to your agent code are required. Use this approach when: * Your agent is built with **LangGraph**. * You want the finest level of trace detail: per-LLM-call, per-tool-call, and per-agent-node spans with structured metadata. * You prefer decorator-based instrumentation over manually writing tracer and exporter code. For an approach that requires no SDK dependency and uses the standard OpenTelemetry SDK instead, see [Exporting observability traces with OpenTelemetry](otel-export). ## How it works The SDK wraps the same OpenTelemetry export pipeline as the [OpenTelemetry export approach](otel-export), but manages it for you: ``` Your LangGraph agent │ ├─ @configure_tracing ← wires tracing context into the compiled graph ├─ @trace_agent_call ← root span per agent node invocation │ ├─ @trace_llm_call ← child span per LLM invocation │ └─ @trace_tool_call ← child span per tool invocation │ └─ @trace_call ← child span for any helper function │ └─ SDK (internal) ├─ Client (authenticates using api_key + instance_url) ├─ TracerProvider + DynamicAuthOTLPSpanExporter └─ ──OTLP/HTTP──► watsonx Orchestrate ingestion endpoint └─► Analytics dashboard Overview / Conversations ``` The setup follows four steps: 1. Initialize a `Client` with your API key and instance URL. 2. Create a `TracerConfig` that identifies the agent and workspace that the exported spans belong to, by using the registered agent's `agent_id`, `workspace_id`, and environment. On AWS deployments, the SDK derives the tenant context automatically from the JWT token. On IBM Cloud, pass `tenant_id` explicitly because the IAM token does not contain tenant information. 3. Build a `Tracer` from the configuration and register it globally with `register_tracer()` to make it available to all SDK decorators. 4. Apply decorators to the relevant functions. Each decorator is a Python `@` annotation that intercepts function calls at runtime to create and export spans without modifying your function logic. After watsonx Orchestrate receives the spans, they appear in **Analyze** > **\[Agent name]** > **Overview** and **Conversations**, showing total conversations, token counts, average duration, and the usage trend chart. ## Before you begin Complete the following setup before you instrument your agent: 1. Import the LangGraph agent through the watsonx Orchestrate UI to create an agent entry and obtain a valid Agent ID. For more information, see [Importing LangGraph agents](https://www.ibm.com/docs/en/watsonx/watson-orchestrate/base?topic=agents-importing-langgraph). 2. Copy the **Agent ID** from the registered agent. 3. For IBM Cloud: Copy the **Workspace ID**. Call the [List workspaces](https://developer.ibm.com/apis/catalog/watsonorchestrate--custom-assistants/api/API--watsonorchestrate--workspace-and-account-management#listWorkspaces) API to fetch the workspaces from a tenant. 4. Copy the **Tenant ID** for your instance. The Tenant ID has the format `_`. Obtain it in one of the following ways: * From the instance CRN in **Profile** > **About**. * From the browser DevTools console, run `WO_Meta` to retrieve its properties. When you use the SDK on AWS deployments, the tenant context is derived automatically from the JWT token inside `Client`. On IBM Cloud, the IAM token does not contain tenant information, so you must pass the Tenant ID explicitly as `tenant_id` in `TracerConfig`. 5. Obtain an **API key** and the corresponding **IAM or MCSP token URL** that is authorized to call the trace ingestion endpoint. For more information, see [Authenticating to the API](https://www.ibm.com/docs/en/watsonx/watson-orchestrate/base?topic=api-getting-started#api-key-getting-started__authenticating-to-the-api__title__1). 6. Note the **instance URL** for your watsonx Orchestrate instance. This URL is the base URL of the ingestion endpoint and has the format `https://api.watson-orchestrate.ibm.com/instances/`. For more information, see [Getting the API endpoint](https://www.ibm.com/docs/en/watsonx/watson-orchestrate/base?topic=api-getting-endpoint). Save the Agent ID, Tenant ID, API key, token URL, and instance URL values for use in the procedure. ## Procedure Because the SDK is currently a pre-release build, install it from Test PyPI. To perform a quick install, run: ```bash BASH theme={null} pip install -i https://test.pypi.org/simple/ ibm-watsonx-orchestrate-sdk ``` For a project with a `requirements.txt`, add the following entries: ```text wrap theme={null} # requirements.txt # Resolve all other packages from public PyPI --extra-index-url https://pypi.org/simple --index-url https://test.pypi.org/simple/ # Add your agent's dependencies here, for example: requests>=2.31.0 # WXO Observability SDK (pre-release, Test PyPI) ibm-watsonx-orchestrate-sdk @ https://test-files.pythonhosted.org/packages/70/8a/04bfbb0e81c369023cbb8a562d81d69c9df27b776b6488a86b55da8d1545/ibm_watsonx_orchestrate_sdk-2.14.0.dev7756-py3-none-any.whl ``` Then install the dependencies: ```bash BASH theme={null} pip install -r requirements.txt ``` At module load time, create a `Client`, build a `TracerConfig`, and register the tracer globally. Complete this step **once**, before any decorated functions are called. ```python Python wrap theme={null} from ibm_watsonx_orchestrate_sdk.client import Client from ibm_watsonx_orchestrate_sdk.observability import ( Tracer, TracerConfig, register_tracer, ) # Authenticate and identify the WXO instance client = Client( api_key="", instance_url="https://api.watson-orchestrate.ibm.com/instances/", ) # Identify the agent and workspace this tracer belongs to # On AWS, tenant_id is extracted automatically from the JWT token inside client config = TracerConfig( client=client, agent_id="", # Agent ID from registration step workspace_id="", environment="live", # "draft" or "live" ) tracer = Tracer(config) register_tracer(tracer) ``` ```python Python wrap theme={null} from ibm_watsonx_orchestrate_sdk.client import Client from ibm_watsonx_orchestrate_sdk.observability import ( Tracer, TracerConfig, register_tracer, ) # Authenticate and identify the WXO instance client = Client( api_key="", instance_url="https://api.watson-orchestrate.ibm.com/instances/", ) # On IBM Cloud, the IAM token does not contain tenant information, # so tenant_id must be provided explicitly in TracerConfig. config = TracerConfig( client=client, tenant_id="", # Required on IBM Cloud; format: _ agent_id="", # Agent ID from registration step workspace_id="", environment="live", # "draft" or "live" ) tracer = Tracer(config) register_tracer(tracer) ``` Key points: * Pass `api_key` and `instance_url` to `Client`. * On AWS, the SDK derives the tenant context automatically from the JWT token, so `tenant_id` is not required. * On IBM Cloud, the IAM API key generates a user-level token that does not include tenant information. Pass `tenant_id` explicitly in `TracerConfig` to avoid a `ValueError: Trace injection mode requires the following parameters: tenant_id` at startup. * The only valid values for `environment` are `"draft"` and `"live"`. * `Tracer` uses `DynamicAuthOTLPSpanExporter` to refresh tokens automatically. Long-running agent processes do not need to be restarted when an access token expires. Apply the SDK decorators to the functions that make up your agent. Manual span creation is not required. The following table shows which decorator to apply and where: | Decorator | Apply to | What it captures | | -------------------- | ------------------------------------------------- | ----------------------------------------------------- | | `@configure_tracing` | The factory function that builds the `StateGraph` | Wires tracing context through the compiled graph | | `@trace_agent_call` | The top-level LangGraph node function | Root span for the agent decision step | | `@trace_llm_call` | The function that calls the LLM | LLM invocation with model, provider, and token counts | | `@trace_tool_call` | Functions also decorated with `@tool` | Tool invocations with tool-specific metadata | | `@trace_call` | Any other helper function | Input, output, and custom attributes | #### General-purpose helper calls: `@trace_call` Use `@trace_call` for helper logic such as validation or response formatting. Set `capture_input=True` or `capture_output=True` to record function arguments and return values as span attributes. ```python Python wrap theme={null} from ibm_watsonx_orchestrate_sdk.observability.decorators import trace_call @trace_call( name="validate_currency", capture_input=True, capture_output=True, attributes={"validation_type": "currency_support"}, ) def validate_currency(currency_code: str) -> tuple[bool, str]: ... ``` #### Tool calls: `@trace_tool_call` Use `@trace_tool_call` on functions that are also decorated with LangChain's `@tool` decorator. The SDK records each tool invocation as a distinct span with tool-specific metadata. `@tool` must be the outermost decorator. Place `@tool` above `@trace_tool_call` in the source file, with `@trace_tool_call` applied directly to the function. ```python Python wrap theme={null} from langchain_core.tools import tool from ibm_watsonx_orchestrate_sdk.observability.decorators import trace_tool_call @tool @trace_tool_call( name="get_exchange_rate_tool", capture_input=True, capture_output=True, tool_name="get_exchange_rate", attributes={"api": "frankfurter", "version": "v2"}, ) def get_exchange_rate(base_currency: str, quote_currency: str) -> str: ... ``` #### LLM calls: `@trace_llm_call` Wrap the function that invokes the LLM with `@trace_llm_call`. Set `model` and `provider` so that the analytics view correctly attributes token and latency metrics. ```python Python wrap theme={null} from ibm_watsonx_orchestrate_sdk.observability.decorators import trace_llm_call @trace_llm_call( name="llm_call", capture_input=True, capture_output=True, model="openai/gpt-4o-mini", provider="openai", attributes={"agent_type": "currency_exchange"}, ) def _invoke_llm(llm, messages): """Invoke the LLM with tracing.""" return llm.invoke(messages) ``` #### Agent node: `@trace_agent_call` Apply `@trace_agent_call` to the top-level LangGraph node that represents the agent decision step. The `agent_name` parameter labels this trace in the analytics view. The `agent_id` in `TracerConfig` identifies the watsonx Orchestrate agent instance to which the trace belongs. ```python Python wrap theme={null} from ibm_watsonx_orchestrate_sdk.observability.decorators import trace_agent_call from langgraph.graph import StateGraph from langchain_core.runnables import RunnableConfig @trace_agent_call( name="agent_node", agent_name="my-external-agent", framework="langgraph", capture_input=True, capture_output=True, attributes={"agent_type": "currency_exchange", "has_tools": "true"}, ) def agent_node(state: AgentState, config: RunnableConfig) -> AgentState: ... ``` #### Graph factory: `@configure_tracing` Apply `@configure_tracing` to the factory function that builds and returns the LangGraph `StateGraph`. This decorator propagates tracing context through the compiled graph. ```python Python wrap theme={null} from ibm_watsonx_orchestrate_sdk.observability.decorators import configure_tracing @configure_tracing def create_agent(config: RunnableConfig) -> StateGraph: workflow = StateGraph(AgentState) workflow.add_node("agent", agent_node_with_config) workflow.add_node("tools", ToolNode([get_exchange_rate])) workflow.add_edge(START, "agent") workflow.add_conditional_edges( "agent", should_continue, {"tools": "tools", "end": END} ) workflow.add_edge("tools", "agent") return workflow ``` ## Result After the decorated agent runs, the SDK exports the captured spans to watsonx Orchestrate by using the `Client` credentials and the configured `agent_id` and tenant context. The traces are visible in watsonx Orchestrate under **Analyze** > **\[Agent name]** > **Overview** and **Conversations**. Per-call span detail is available in the Conversations drill-down view. ## Troubleshooting ##### Verifying trace ingestion If traces do not appear on the Analytics page, use the [Get Traces API](https://developer.ibm.com/apis/catalog/watsonorchestrate--custom-assistants/api/API--watsonorchestrate--agentops-and-analytics#getTraces) to confirm whether the ingestion endpoint received them. If the API returns your traces, ingestion was successful. If the Get Traces API returns an empty list, the spans were not received. Check the following items: * **`401 Unauthorized` on startup**: The API key or instance URL passed to `Client` is incorrect. Verify that both values match the watsonx Orchestrate instance where the agent is registered. * **Token expiry during a long run**: The `DynamicAuthOTLPSpanExporter` refreshes tokens automatically. If authentication failures persist, confirm that the API key has not been revoked and that the instance URL is reachable from the agent host. * **`agent_id` or `workspace_id` not found**: The values passed to `TracerConfig` must match the Agent ID and Workspace ID shown in the watsonx Orchestrate UI for the registered agent. Verify both values against the registration entry. * **`@configure_tracing` not applied**: If `@configure_tracing` is missing from the graph factory function, tracing context is not propagated through the compiled graph, and child spans may not be associated with the correct root span.