> ## Documentation Index
> Fetch the complete documentation index at: https://developer.watson-orchestrate.ibm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluating agents and tools

## Before you begin

In order to evaluate, you must first have your ground truth dataset defined. To create your dataset, see [Creating evaluation dataset](./create_data).

<Note>
  **Note:**

  * Starting from version 1.12.0, this command now works with SaaS and on-premises offerings of watsonx Orchestrate. You can now run your evaluations in remote instances, instead of just using the watsonx Orchestrate Developer Edition.

  * If you're using Inference Frameworks Manager (IFM) in an on-premises environment, you likely need to set the MODEL\_OVERRIDE environment variable to enable the evaluation feature. For more information, see [Evaluations on CPD](https://github.com/IBM/ibm-watsonx-orchestrate-adk/blob/main/examples/evaluations/cpd/README.md).
</Note>

## Evaluating

The `evaluate` command lets you test and benchmark your agents using ground truth datasets. These datasets can be [created automatically](./create_data) (with the `generate` or `record` commands) or prepared manually. The evaluation process measures your agent's performance against the expected outcomes defined in these datasets.

* The system simulates user interactions described in your datasets.
* The agent's responses are compared step-by-step with the ground truth dataset.
* Any mistakes or deviations from the expected trajectory are logged.
* At the end, a summary table of metrics is displayed and saved as a CSV.

### Prerequisites

1. You must import all **agents** and **tools** before running the evaluation.
2. Ensure that you have set up the environment properly. For more information, see [Configuring your environments](../environment/initiate_environment).
3. Configure a `.env` file to provide authentication details to the evaluation framework. Most of the commands support a `--env (-e)` command, so you can pass your `.env` file. See an example of a `.env` file:

   ```env .env theme={null}
   WO_INSTANCE=your watsonx Orchestrate instance URL
   WO_API_KEY=your watsonx Orchestrate API key
   WO_USERNAME=your watsonx Orchestrate username (only for on-premises)
   WO_PASSWORD=your watsonx Orchestrate user password (only for on-premises, and if API key is not provided)
   ```

   In on-premises environments you can also use the following environment variables:

   ```env .env theme={null}
   WO_SSL_VERIFY=false   # to avoid self-signed certificate issues when authenticating to the server
   WATSONX_SPACE_ID=space ID for watsonx.ai runtime instance   # use it if you are using watsonx.ai models instead of Cloud Pak for Data's Inferencing Foundation Models (IFM)
   WATSONX_APIKEY=watsonx.ai API key   # not required
   MODEL_OVERRIDE=model name   # If your IFM environment only exposes a single model, use it to force that model at runtime.
   ```

   For more information, see the [on-premises ADK examples](https://github.com/IBM/ibm-watsonx-orchestrate-adk/blob/main/examples/evaluations/cpd).

### Agentops evaluation

<Check>New in 2.1.0</Check>

The `evaluate` command supports evaluation with **Langfuse**.

1. To enable it, you must set the following environment variable:

   ```env theme={null}
   USE_LEGACY_EVAL=FALSE
   ```

   <Warning>
     Other evaluation commands might not work properly with this environment variable set to `FALSE`. For example, if you use the `analyze` command, the outputs will not be automatically created.

     To safely disable this behavior, and make other evaluation commands work normally, you must set the environment variable value to `TRUE`:

     ```
     USE_LEGACY_EVAL=TRUE
     ```
   </Warning>

2. Before you run the evaluation, you must start the watsonx Orchestrate Developer Edition server with [**Langfuse**](../llm/observability):

   ```bash BASH theme={null}
   orchestrate server start --env-file <ENV FILE> -l
   ```

   After the server starts, you can access your local Langfuse dashboard at `http://localhost:3010`.

   Your Langfuse username and password will appear in the terminal. Use these credentials to log in to the dashboard.

   To learn more about the watsonx Orchestrate Developer Edition, see [Installing watsonx Orchestrate Developer Edition](../developer_edition/wxOde_setup).

   To learn more about Langfuse, see [Monitoring your LLMs with Langfuse](../llm/observability).

You can proceed to run the [`evaluate`](#evaluate) command. The results of the evaluation will appear in the **Langfuse** dashboard.

### `evaluate`

To evaluate your agent, use:

```bash BASH theme={null}
orchestrate evaluations evaluate --test-paths path1,path2 --output-dir output_directory
```

You can also run evaluation using a YAML config file, giving you full control over all parameters.

```bash BASH theme={null}
orchestrate evaluations evaluate --config path/to/config.yaml
```

**Sample config file:**

<CodeGroup>
  ```yaml Local theme={null}
  test_paths:
    - benchmarks/wxo_domains/rel_1.8_mock/workday/data/
  auth_config:
    url: http://localhost:4321
    tenant_name: local
  output_dir: "test_bench_data3"
  enable_verbose_logging: true
  llm_user_config:
    user_response_style:
    - "Be concise in messages and confirmations"
  n_runs: 2   # evaluations will run 2 times
  ```

  ```yaml SaaS theme={null}
  test_paths:
    - benchmarks/wxo_domains/rel_1.8_mock/workday/data/
  auth_config:
    url: https://api.<my-region>.watson-orchestrate.ibm.com/instances/<your-instance> # substitute with your API URL
    tenant_name: saas # substitute with the name of the environment
  output_dir: "test_bench_data3"
  enable_verbose_logging: true
  llm_user_config:
    user_response_style:
    - "Be concise in messages and confirmations"
  wxo_lite_version: 1.12.0
  n_runs: 2   # evaluations will run 2 times
  ```

  ```yaml On-premises theme={null}
  test_paths:
    - benchmarks/wxo_domains/rel_1.8_mock/workday/data/
  auth_config:
    url: https://<my-api-url>:<port>/orchestrate/instances/<your-instance> # substitute with your API URL
    tenant_name: onprem # substitute with the name of your environment
  output_dir: "test_bench_data3"
  enable_verbose_logging: true
  llm_user_config:
    user_response_style:
    - "Be concise in messages and confirmations"
  wxo_lite_version: 1.12.0
  n_runs: 2   # evaluations will run 2 times
  ```
</CodeGroup>

* Evaluations can be repeated multiple times to assess performance robustness. This is configured by setting the `n_runs` parameter in the YAML configuration file. If `n_runs` is not specified, the evaluation defaults to a single run.
* Tool call arguments are automatically normalized during evaluation. This ensures that minor differences in formatting — such as capitalization, key order, value types, or list order — no longer result in mismatches.

<Note>
  **For SaaS and On-premises only:**\
  On `tenant_name`, use the name of the environment that you used when you added the environment with the `orchestrate env add` command.
</Note>

<Accordion title="Flags">
  <ParamField path="--config (-c)" type="string">
    Path to the configuration file with details about the datasets and the output directory.
  </ParamField>

  <ParamField path="--test-paths (-p)" type="list[string]">
    Comma-separated list of test files or directories containing ground truth datasets.
  </ParamField>

  <ParamField path="--output-dir (-o)" type="string">
    Directory where evaluation results will be saved.
  </ParamField>

  <ParamField path="--env-file (-e)" type="string">
    Path to the `.env` file that overrides the default environment.
  </ParamField>
</Accordion>

<Accordion title="Configuration File Reference">
  The following fields can be used in your `config.yaml` file for the `evaluate` command.

  ### Required Fields

  <ParamField path="test_paths" type="List[str]">
    Paths to test files/directories to evaluate
  </ParamField>

  <ParamField path="output_dir" type="str">
    Directory where evaluation results will be stored
  </ParamField>

  ### Optional Fields

  #### Authentication & Provider

  <ParamField path="auth_config" type="AuthConfig">
    Authentication configuration

    * `url` (str) - Service URL
    * `tenant_name` (str) - Tenant/environment name (default: "local")
    * `token` (str) - Authentication token
  </ParamField>

  <ParamField path="provider_config" type="ProviderConfig">
    LLM provider configuration

    * `provider` (str) - Provider type: "gateway", "watsonx", or "model\_proxy" (default: "gateway")
    * `model_id` (str) - Model ID to use
    * `embedding_model_id` (str) - Embedding model
    * `vendor` (str) - Vendor name (default: "ibm")
    * `referenceless_eval` (bool) - Enable referenceless evaluation (default: false)
    * `provider_params` (dict) - Additional provider parameters
  </ParamField>

  #### LLM User Simulation

  <ParamField path="llm_user_config" type="LLMUserConfig">
    User simulator configuration

    * `model_id` (str) - Model for user simulation
    * `prompt_config` (str) - Path to prompt template
    * `user_response_style` (List\[str]) - User response style guidelines
    * `personas` (List\[str]) - User personas to simulate
    * `enable_structured_output` (bool) - Enable structured output (default: false)
    * `version` (str) - Version: "v1" or "v2" (default: "v1")
  </ParamField>

  #### Evaluation Configuration

  <ParamField path="max_user_turns" type="int">
    Maximum conversation turns (default: 20)
  </ParamField>

  <ParamField path="n_runs" type="int">
    Number of evaluation runs (default: 1)
  </ParamField>

  <ParamField path="num_workers" type="int">
    Number of parallel workers (default: 2)
  </ParamField>

  <ParamField path="skip_legacy_evaluation" type="bool">
    Skip legacy evaluation (default: false)
  </ParamField>

  <ParamField path="skip_available_results" type="bool">
    Skip existing results (default: false)
  </ParamField>

  <ParamField path="data_annotation_run" type="bool">
    Run in annotation mode (default: false)
  </ParamField>

  <ParamField path="evaluation_model" type="str">
    Model for evaluation
  </ParamField>

  #### Metrics & Operators

  <ParamField path="metrics" type="List[str]">
    Metrics to compute (default: \["JourneySuccessMetric", "ToolCalling", "OrchestrateAgentRoutingAccuracy", "StepMetrics", "AgentResponseTime"])
  </ParamField>

  <ParamField path="operator_configs" type="dict">
    Configuration for each metric operator
  </ParamField>

  <ParamField path="custom_metrics_config" type="CustomMetricsConfig">
    Custom metrics configuration

    * `paths` (List\[str]) - Paths to custom metric modules
    * `llmaaj_config` (ProviderConfig) - LLM-as-a-judge configuration
  </ParamField>

  #### Text Matching & Validation

  <ParamField path="similarity_threshold" type="float">
    Similarity threshold (default: 0.8)
  </ParamField>

  <ParamField path="enable_fuzzy_matching" type="bool">
    Enable fuzzy matching (default: false)
  </ParamField>

  <ParamField path="is_strict" type="bool">
    Strict validation mode (default: true)
  </ParamField>

  <ParamField path="enable_recursive_search" type="bool">
    Enable recursive search (default: false)
  </ParamField>

  <ParamField path="text_match_config" type="TextMatchConfig">
    Text matching configuration

    * `semantic_template_path` (str) - Semantic matching template
    * `keyword_template_path` (str) - Keyword matching template
  </ParamField>

  #### Other Options

  <ParamField path="wxo_lite_version" type="str">
    Version of the WXO lite SDK
  </ParamField>

  <ParamField path="tags" type="List[str]">
    Tags for filtering/organizing tests
  </ParamField>

  <ParamField path="error_keywords" type="List[str]">
    Keywords to identify errors
  </ParamField>

  <ParamField path="enable_verbose_logging" type="bool">
    Enable verbose logging (default: true)
  </ParamField>

  <ParamField path="enable_manual_user_input" type="bool">
    Enable manual user input (default: false)
  </ParamField>

  <ParamField path="collection_name" type="str">
    Collection name (default: "default-collection")
  </ParamField>

  <ParamField path="langfuse_enabled" type="bool">
    Enable Langfuse integration (default: false)
  </ParamField>

  <ParamField path="extractors_config" type="ExtractorsConfig">
    Extractors configuration

    * `paths` (List\[str]) - Paths to extractor modules
  </ParamField>
</Accordion>

After any evaluation run, a `results/config.yml` file is generated, capturing all used parameters. This can serve as a template for future runs.

If you enabled the [Agentops evaluation](#agentops-evaluation), the results will guide you to your Langfuse dashboard:

```
Config and metadata saved to output/evaluate_1110/2025-11-10_21-06-55
Langfuse Evaluation run completed for collection 2025-11-10_21-06-55_collection:
 - http://localhost:3010/project/orchestrate-lite/sessions/e1bac47c-060a-46a7-8820-3260fd5f7252
 - http://localhost:3010/project/orchestrate-lite/sessions/1b56c480-4a2d-45ed-9c55-23e8a71f0a16
```

And the results will appear in your Langfuse dashboard:

<img src="https://mintcdn.com/ibm-2e3153bf/wDf2BNAA3KbRN4Go/images/langfuse-dashboard.png?fit=max&auto=format&n=wDf2BNAA3KbRN4Go&q=85&s=143434782f0692a57df82b751a424856" alt="Example langfuse dashboard" width="1588" height="239" data-path="images/langfuse-dashboard.png" />

## Understanding the Summary Metrics Table

At the end of the evaluation, you will see a summary similar to the following one:

<img src="https://mintcdn.com/ibm-2e3153bf/SjuDH1-1MNHBclq7/images/evaluationtable.png?fit=max&auto=format&n=SjuDH1-1MNHBclq7&q=85&s=25800990365b0ae63a5a5680a58e7020" alt="Summary metrics table" width="1364" height="210" data-path="images/evaluationtable.png" />

This table is also saved as a CSV file at `results/summary_metrics.csv`.

### Metrics explained

**Agent with Knowledge Summary Metrics**

| Metric                             | Description                                                                                         | Calculation / Type |
| ---------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------ |
| **Average Response Confidence**    | Calculates an average of the confidence that the responses actually answer the user's question      | Float (≥ 0.0)      |
| **Average Retrieval Confidence**   | Calculates an average of the confidence that the retrieved document is relevant to the user's query | Float (≥ 0.0)      |
| **Average Faithfulness**           | Calculates an average of how close the responses match the the values in the knowledge base         | Float (≥ 0.0)      |
| **Average Answer Relevancy**       | Calculates how relevant the answers are based on the knowledge base queries                         | Float (≥ 0.0)      |
| **Number Calls to Knowledge Base** | Total number of knowledge bases called                                                              | Integer (≥ 0)      |
| **Knowledge Bases Called**         | Names of the knowledge bases that are called                                                        | Text               |

**Agent Metrics**

| Metric                     | Description                                                                                                                        | Calculation / Type  |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------- |
| **Runs**                   | Number of evaluation runs for the dataset, as defined by the `n_runs` parameter.                                                   | Integer (≥ 1)       |
| **Total Steps**            | Total messages/steps in the conversation                                                                                           | Integer (≥ 0)       |
| **LLM Steps**              | Assistant responses (text/tool calls)                                                                                              | Integer (≥ 0)       |
| **Total Tool Calls**       | Total number of tool calls made                                                                                                    | Integer (≥ 0)       |
| **Tool Call Precision**    | Calculates the number of correct tool calls divided by the total number of tool calls made                                         | Float (≥ 0.0)       |
| **Tool Call Recall**       | Determines if the agent called the right tools in the right order.                                                                 | Float (≥ 0.0)       |
| **Agent Routing Accuracy** | Determines if the agent reroutes to the expected agents. If there's no agent routing in the simulation, the default value is `0.0` | Integer (≥ 0)       |
| **Text Match**             | Determines if the final response is similar and accurate to the expected response                                                  | Percentage (0–100%) |
| **Journey Success**        | Considers if the agent made the agent calls in the correct order, matching all the established criteria in the `goal_details`      | Boolean             |
| **Avg Resp Time (Secs)**   | Average response time for agent responses                                                                                          | Float (≥ 0.0)       |

<Note>
  If the value is equal to 1.0 or `True`, the table omits the result.
</Note>

### Per-Dataset Detailed Results

In the `results/messages` directory, you will find detailed analysis files for each dataset.

For single-run evaluations, the following files are generated:

* `<DATASET>.messages.json`: Raw messages exchanged during simulation.
* `<DATASET>.messages.analyze.json`: Annotated analysis, including mistakes and step-by-step comparison to ground truth.
* `<DATASET>.metrics.json`: Metrics summary for that specific test case.

For multi-run evaluations (when the n\_runs parameter is set to more than 1), separate files are generated for each run using a run-indexed naming pattern:

* `<DATASET>.run<runN>.messages.json`
* `<DATASET>.run<runN>.messages.analyze.json`
* `<DATASET>.run<runN>.metrics.json`

Here, `<runN>` represents the run number (for example, run1, run2, and so on).

<Tip>
  * Always verify that your API credentials are set before running `evaluate`.
  * Review per-dataset result files for deep insight into agent performance and error patterns.
  * Tune config parameters as needed for different evaluation scenarios.
  * Use summary metrics for quick benchmarking, but always check details for full understanding.
</Tip>

<Note>
  **Note:**

  You evaluate an external agent's performance the same way as a native agent. When you review the final summary table from the `evaluate` command, focus only on the `Text Match` and `Journey Success` columns. Native tool calls aren't involved, so no other columns apply.
</Note>