> ## Documentation Index
> Fetch the complete documentation index at: https://developer.watson-orchestrate.ibm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Quick evaluation of agents and tools

The `quick-eval` command provides a fast, reference-less evaluation of your agents and tools.

<Note>
  **Note:**

  For now, you can use only Python tools.
</Note>

Unlike the standard `evaluate` command, it does not require ground truth datasets. Instead, it runs a lightweight check to identify common issues such as schema mismatches and hallucinations in tool calls.

```bash BASH theme={null}
orchestrate evaluations quick-eval -p  examples/evaluations/quick-eval/ -o results/ -t examples/evaluations/evaluate/agent_tools
```

You can also run the quick evaluation using a YAML config file, giving you full control over all parameters.

```bash BASH theme={null}
orchestrate evaluations quick-eval -c examples/evaluations/config.yaml
```

**Sample config file:**

```yaml config.yaml theme={null}
test_paths:
  - benchmarks/wxo_domains/rel_1.8_mock/workday/data/
auth_config:
  url: http://localhost:4321
  tenant_name: local
output_dir: "test_bench_data3"
enable_verbose_logging: true
llm_user_config:
  user_response_style:
  - "Be concise in messages and confirmations"
```

<Accordion title="Flags">
  <ParamField path="--config (-c)" type="string" required>
    Path to the configuration file with details about the evaluation settings.
  </ParamField>

  <ParamField path="--test-paths (-p)" type="list[string]">
    Comma-separated list of test files or directories containing ground truth datasets. Required when not using a configuration file.
  </ParamField>

  <ParamField path="--tools-path (-t)" type="string">
    Directory containing tool definitions.
  </ParamField>

  <ParamField path="--output-dir (-o)" type="string">
    Directory where evaluation results will be saved. Required when not using a config file.
  </ParamField>

  <ParamField path="--env-file (-e)" type="string">
    Path to the `.env` file that overrides the default environment.
  </ParamField>
</Accordion>

**More examples in the [Examples folder](https://github.com/IBM/ibm-watsonx-orchestrate-adk/tree/main/examples/evaluations/quick-eval)**.

## Understanding the Summary Metrics Table

At the end of the evaluation, you will see a summary similar to the following one:

<img src="https://mintcdn.com/ibm-2e3153bf/D7Bnf9WVmmmv204S/images/quick_eval_results.png?fit=max&auto=format&n=D7Bnf9WVmmmv204S&q=85&s=fa87ae0ae697d33db4bf815a6d7d1d58" alt="Quick evaluation results table" width="2914" height="374" data-path="images/quick_eval_results.png" />

### Metrics explained

**Quick Evaluation Summary Metrics**

| Metric                                       | Description                                                                                           | Calculation / Type |
| -------------------------------------------- | ----------------------------------------------------------------------------------------------------- | ------------------ |
| **Dataset**                                  | Name of the dataset used for quick evaluation                                                         | Text               |
| **Tool Calls**                               | Total number of tool calls attempted during the evaluation                                            | Integer (≥ 0)      |
| **Successful Tool Calls**                    | Number of tool calls that executed successfully without errors                                        | Integer (≥ 0)      |
| **Tool Calls Failed due to Schema Mismatch** | Number of tool calls that failed because the input/output schema did not match expectations           | Integer (≥ 0)      |
| **Tool Calls Failed due to Hallucination**   | Number of tool calls that failed because the agent invoked tools that were irrelevant or non-existent | Integer (≥ 0)      |

<Note>
  If the value is equal to 1.0 or `True`, the table omits the result.
</Note>