Creating test cases for evaluation

There are two primary methods for generating test cases to evaluate your agents:

Using the record command Captures live chat sessions via the chat UI and converts them into test cases. Useful for benchmarking agent behavior across different configurations (e.g., agent descriptions, LLMs).
Using the generate command Automatically creates test cases from user stories and tool definitions. Ideal for building realistic and repeatable evaluation scenarios.

Recording user interaction

The record command captures real-time chat interactions and automatically generates test cases from them. With recording enabled, any conversation you have in the chat UI will automatically be captured and annotated for evaluation. You can create as many test cases as you want. Start a new chat session in the preview panel for each one.

Tip:
Create a new chat session when you want to make new test cases. Using the same chat session for multiple tests can cause issues with the final output.

Note:When you work with external agents that act as collaborators of native agents, use the record command the same way you do with native agents. The key difference is that the generated ground truth data doesn’t include a graph of tool calls in the "goals" and "goal_details" sections.

Show example

JSON

{
    "agent": "NATIVE_AGENT_NAME",
    "goals": {
        "summarize": [
        ]
    },
    "goal_details": [
        {
            "name": "summarize",
            "type": "text",
            "response": "FINAL RESPONSE MESSAGE",
            "keywords": [
                "KEYWORD_1",
                "KEYWORD_2",
                "..."
            ]
        }
    ],
    "story": "STORY OF THE INTERACTION",
    "starting_sentence": "STARTING SENTENCE USED TO START THE INTERACTION"
}

Workflow

Interact with the agent via the chat UI.
Use the record command to capture the session.
The session is converted into a test case for evaluation.

Prerequisites

Activate your environment

You must activate an environment before you record your data.For more information about how to add an environment, see Configure access to remote environments.

BASH

orchestrate env activate

Launch the Chat UI

SaaS and On-premises

Open the watsonx Orchestrate chat URL. For example:

https://<my-region>.watson-orchestrate.ibm.com/chat

https://<my-region>.watson-orchestrate.cloud.ibm.com/chat

# Only example URL, your on-premises environment can use a different format
https://<yourlink>:<port>/orchestrate/chat

Local (watsonx Orchestrate Developer Edition)

Make sure your chat UI is running. Use the following command to start the chat interface:

BASH

orchestrate chat start

Once the UI is running, open your browser and navigate to:http://localhost:3000/chat-liteHere, you can select the agent you wish to interact with. For example, the image below uses the hr_agent agent:

Start recording your session

To begin recording, run the following command in your terminal:

BASH

orchestrate evaluations record --output-dir ./chat_recordings

Show command flags

string

Directory where your recorded data will be saved.
If omitted, the data will be saved in your current working directory.
For every chat session, the following file is generated in your output directory:
<THREAD_ID>_annotated_data.json
Annotated ground truth data based on your chat session, ready for evaluation.

string

Context variables as a JSON string. This allows you to add custom context variables to your recorded sessions that will be included in the generated annotated data.

Using context variables with record mode

You can provide context variables that will be included in your recorded sessions. This is useful for adding metadata or session-specific information that you want to track during evaluation:

BASH

orchestrate evaluations record \
  --output-dir ./chat_recordings \
  --context-variables '{"user_id": "user123", "environment": "production", "session_type": "demo"}'

When context variables are provided, the annotated data will include a context_variables field:

JSON

{
    "agent": "hr_agent",
    "thread_id": "abc123-def456",
    "context_variables": {
        "user_id": "user123",
        "environment": "production",
        "session_type": "demo"
    },
    "goals": {
        "get_assignment_id_hr_agent-1": [
            "get_timeoff_schedule_hr_agent-1"
        ]
    },
    "goal_details": [
        {
            "type": "tool_call",
            "name": "get_assignment_id_hr_agent-1",
            "tool_name": "get_assignment_id_hr_agent",
            "args": {
                "username": "nwaters"
            }
        }
    ],
    "story": "You want to know your time off schedule...",
    "starting_sentence": "I want to know my time off schedule"
}

Tip:
Context variables are particularly useful for:

Tracking different user personas or environments
Adding session metadata for analysis
Grouping related test cases
Maintaining consistency across evaluation runs

Example annotated data

The following is a sample conversation with the hr_agent:

This conversation generates the following annotated data file:

{
    "agent": "hr_agent",
    "goals": {
        "get_assignment_id_hr_agent-1": [
            "get_timeoff_schedule_hr_agent-1"
        ],
        "get_timeoff_schedule_hr_agent-1": [
            "summarize"
        ]
    },
    "goal_details": [
        {
            "type": "tool_call",
            "name": "get_assignment_id_hr_agent-1",
            "tool_name": "get_assignment_id_hr_agent",
            "args": {
                "username": "nwaters"
            }
        },
        {
            "type": "tool_call",
            "name": "get_timeoff_schedule_hr_agent-1",
            "tool_name": "get_timeoff_schedule_hr_agent",
            "args": {
                "assignment_id": "15778303",
                "end_date": "2025-01-30",
                "start_date": "2025-01-01"
            }
        },
        {
            "name": "summarize",
            "type": "text",
            "response": "Your time off schedule is on January 5, 2025.",
            "keywords": [
                "January 5, 2025"
            ]
        }
    ],
    "story": "You want to know your time off schedule. Your username is nwaters. The start date is 2025-01-01. The end date is 2025-01-30.",
    "starting_sentence": "I want to know my time off schedule"
}

Note:

The annotated data is generated automatically. Therefore, it is essential to review and, if necessary, edit the data before using it for evaluation purposes. You can also delete any details that are not relevant to your tests.
The starting_sentence field is populated directly from your inputs. However, other fields like story and goals are derived from the recorded conversation and might require validation to ensure their accuracy and relevance.

Stopping the recording

When you are done with your session, press Ctrl+C in the terminal running the record command. Be sure to finish your conversation before stopping to avoid generating an incomplete test case.

Generating user data

The generate command transforms user stories into structured test cases using your tool definitions. It produces test cases suitable for automated evaluation and benchmarking of agents.

Note:For now, you can use only Python tools.

Key Features

Converts user stories into structured test cases
Generates valid tool call sequences based on your tool definitions
Outputs test cases for consistent and automated agent evaluation

Prerequisites

Before running the generate command, ensure the following:

Tool Definitions: Define tools in a Python module using the @tool decorator and proper type annotations.
User Stories: Prepare a .csv file containing user stories. Each row should include:
- story: A natural language description of the user’s goal
- agent: The name of the agent responsible for handling the story
Environment Setup: Import your tool and agent definitions into the environment where the command will be executed. For more information, see Importing Tools.

Example user stories

story,agent
I want to look up holiday calendar for this year. My email-id is someone@gmail.com,hr_agent

You can find example of stories and tools in the following links:

The `generate` command

You can run the following command to run the command:

orchestrate evaluations generate --stories-path <path-to-stories> --tools-path <path-to-tools>

Show command flags

string

Path to your CSV file of user stories.

string

Path to your Python file defining agent tools.

string

Output directory for generated files;
if omitted, files are saved alongside your stories file.

The generate command will analyze each story and generate a sequence of tool calls which is saved as an <AGENT_NAME>_snapshot_llm.json file in our output directory. The snapshot is then used to generate structured test cases that you can use for evaluating your agent(s). The generated test cases are written to a <AGENT_NAME>_test_cases/ folder in the output directory.

Tool requirements

Tool definitions must be provided in a Python file and must follow these requirements:

Functions must be top-level (not inside classes)
Each tool must use the @tool decorator
Each tool must include a descriptive docstring
Argu.ments must be typed (str, int, etc.)
Return values must be JSON-serializable (str, list, dict, etc.)

Example:

@tool()
def fetch_assignment_id(username: str) -> str:
    """Returns the assignment ID for a given employee username.

    Args:
        username (str): The employee's username.

    Returns:
        str: The assignment ID as a string, or 'not found' if no assignment exists.
    """
    assignment_ids = {
        "nwaters": "15778303",
        "johndoe": "15338303",
        "nken": "15338304"
    }
    return assignment_ids.get(username, "not found")

Note:
The tools provided in this example are mocked and use hardcoded data. If your tools need to make actual API calls, make sure to include the necessary authentication credentials (API keys, tokens, etc.) and proper error handling in your implementation.

Tip:

Always verify that your API credentials are set before running generate.
Generated test cases serve as ground truth for benchmarking and validating agent behavior.

Release Notes

Get Started

Build

Deploy

Analyze

Developer experience

Legal notices

Recording user interaction

Workflow

Prerequisites

Start recording your session

Using context variables with record mode

Example annotated data

Stopping the recording

Generating user data

Key Features

Prerequisites

Example user stories

The `generate` command

Tool requirements

​Recording user interaction

​Workflow

​Prerequisites

​Start recording your session

​Using context variables with record mode

​Example annotated data

​Stopping the recording

​Generating user data

​Key Features

​Prerequisites

​Example user stories

​The generate command

​Tool requirements

Recording user interaction

Workflow

Prerequisites

Start recording your session

Using context variables with record mode

Example annotated data

Stopping the recording

Generating user data

Key Features

Prerequisites

Example user stories

The `generate` command

Tool requirements