Skip to main content
The evaluation framework in the ADK is designed to assess agent behavior by comparing simulated agent interactions, referred to as trajectories, against a predefined set of reference data. It also includes tools to generate this reference data and supports the entire agent development lifecycle.

Setup environment

By default, the evaluation framework uses your localhost development environment (watsonx Orchestrate Developer Edition). You can use a .env file to change your environment with the --env-file flag. To do that, you must set up your .env file similar to how you install the watsonx Orchestrate Developer Edition. Most of the evaluation framework capabilities depend on model-proxy or wx.ai models. If you use the Dallas data center, everything works fine. But if you need to run the framework with model-proxy and a WO_INSTANCE that points to a non-Dallas region, you can supply the model override flag:
The evaluation framework works with SaaS and on-premises offerings of watsonx Orchestrate. You can now run your evaluations only in your draft environments of the remote instances. To do that, you must activate your environment and run the evaluation commands on this active remote environment. For more information see Configure access to remote environments.

Agent Pre-Deployment Lifecycle

The agent development process before deployment is iterative and consists of four key stages:
  1. Develop: Build and configure the agent.
  2. Evaluate: Measure performance against reference data.
  3. Analyze: Inspect agent behavior and identify patterns or issues.
  4. Improve: Refine the agent based on evaluation results.

Evaluation process

Agent evaluation is driven by a user story, which provides all necessary context for the agent to perform tool-based tasks. A user story consists of all the necessary context and goals that the user expects to achieve in a specific interaction with the agent. A good user story includes user information, as well as the description of the goal and overall context of the interaction. During the evaluation process, the evaluation framework simulates a user interaction with the target agent, which is the agent being evaluated. This new interaction is driven by a user agent that is powered by an LLM and follows the instructions given in the user story. The user agent uses the context of the user story to send messages to the target agent. The responses returned by the target agent are compared to the expected responses in the user story, and if they are an exact match, the target agent successfully passes the evaluation.

Success criteria

A trajectory is a sequence of actions taken by an agent in response to a user query. A trajectory is considered successful when:
  • The agent performs all required tool calls in the correct order with the expected input parameters.
  • The agent generates a high-quality summary that effectively addresses the user’s query.
When both conditions are met, the evaluation framework marks the interaction as a successful journey, contributing to the Journey Success metric.

Ground truth datasets

Ground truth datasets are used to benchmark and evaluate agent performance. These datasets are structured in JSON format and include:
  • Full conversation context
  • Tool call sequences
  • Dependency graphs
  • Expected final summaries
The evaluation framework uses this data to score agent behavior and identify areas for improvement. Example dataset
  • agent
    • Type: string
    • Description: The name of the agent involved in the conversation. This is used to associate the dataset with a specific agent. To find the agent name, you can call orchestrate agents list
  • goals
    • Type: object
    • Description: Represents a goal dependency graph (a map of goal identifiers to arrays of dependent goals.). This encodes the logical or causal order in which tool calls should occur. Each key is a unique goal name or step in the process (e.g., "get_assignment_id_hr_agent-1"). Each value is an array of goals that depend on completion of this goal.
    Example:
    This means that the agent must get the Assignment ID before it gets the timeoff schedule.
  • goal_details
    • Type: array
    • Description: A step-by-step list of actions the agent must take. Each object includes details about the action type, name, and type-specific fields.
    Action Types:
    • "tool_call": A specific tool call, with arguments specified in args.
    • "text": A text response, including the full response and key terms in keywords.
    • "assert_llm": An LLM-as-a-judge assertion. The LLM judge reads the conversation and determines whether the check statement is true or false. For more information, see Assertions.
    Fields for each action:
    • type: "tool_call", "text", or "assert_llm"
    • name: Action or step name (unique in each dataset; not used for assert_llm)
    • tool_name: (For tool_call) Name of the tool/function invoked
    • args: (For tool_call) Dictionary of arguments passed to the tool
    • response: (For text) The agent’s natural language response
    • keywords: (For text) Key terms in the response
    • kind: (For assert_llm) The assertion kind. Currently only "observed" is supported.
    • scope: (For assert_llm) The portion of the conversation the judge reads: "response" (last agent message only) or "conversation" (full conversation, default).
    • check: (For assert_llm) The plain-language statement the judge evaluates as true or false.
  • story
    • Type: string
    • Description: A free-text narrative summary of the user’s intent, key variables, and the main storyline of the data.
  • starting_sentence
    • Type: string
    • Description: The first user utterance or the main initiating statement of the chat session.

Assertions

An assertion is a statement about an agent conversation that is either true or false. You write each assertion in plain language as part of a goal_detail with "type": "assert_llm". An LLM judge reads the conversation and decides whether the statement is true. The judge returns a pass or a fail and one sentence of reasoning. If the conversation does not contain enough information to evaluate the statement, the assertion fails. Each assertion has a kind and a scope: The metric score for assertions is the ratio of passed assertions to total assertions. A test case is a success only when all of its assertions pass.
Assertion goals are the preferred method for adding LLM-as-a-judge checks to your test case over text summary steps. Assertion goals are not added to the goals dependency graph.

Argument Matching Strategies

The framework supports multiple argument matching strategies for tool call validation. Use the arg_matching field in a tool_call goal to specify a strategy per argument. If arg_matching is omitted, all arguments default to strict.

Strict matching (default)

Requires the argument value to match exactly. This is the default when arg_matching is not specified.

Fuzzy matching

Uses semantic similarity to compare argument values. Useful for natural language fields where the exact wording may vary but the meaning is the same.
The fuzzy matcher uses the following fallback chain:
  1. Primary: Cosine similarity over embeddings (default model: sentence-transformers/all-minilm-l6-v2)
  2. Fallback: Token similarity score via fuzzywuzzy.WRatio
You can control the similarity threshold and enable fuzzy matching globally in your config.yml:

Optional matching

The argument is skipped if it is absent from the actual tool call. If it is present, it must match exactly.

Ignore matching

Skips a specific argument entirely during validation. Use this for runtime-generated values such as session tokens, folder IDs, or other fields that are non-deterministic and do not affect correctness.

Ignoring all arguments

To skip argument validation entirely for a tool call, set args to {"IGNORE": null} and omit arg_matching. The framework treats this as an unconditional pass for the argument check — only the tool name is verified.
Use {"IGNORE": null} when you only care that a tool was called, not what arguments it received. This is useful for tools whose arguments are entirely runtime-generated.

Using the <IGNORE> placeholder value

As an alternative to the ignore strategy, you can set an individual argument’s expected value to the string "<IGNORE>". The framework skips the comparison for any field whose expected value is "<IGNORE>", regardless of the strategy set in arg_matching.
This approach is equivalent to setting "session_id": "ignore" in arg_matching, but keeps the placeholder visible directly in args for documentation purposes.
Prefer the "ignore" strategy in arg_matching for new datasets. The "<IGNORE>" value is supported for backwards compatibility.

Nested argument paths

When a tool accepts a Pydantic model or nested object as an argument, you can use dot-notation in arg_matching to target individual nested fields.
You can also apply a single strategy to an entire nested object by specifying its top-level key:
This applies fuzzy matching to all fields inside input.

Generating datasets

You don’t need to create the datasets manually. The ADK includes commands to generate them automatically depending on your use case. For more information, see Creating evaluation dataset.