> ## Documentation Index
> Fetch the complete documentation index at: https://developer.watson-orchestrate.ibm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# wxo-validate-test-case skill

> [!TIP]
>
> New in 2.20.0

The `wxo-validate-test-case` skill validates unified test case JSON files against a live watsonx Orchestrate agent. It directs the agent through the tool chain that the test case declares, and classifies any failures.

**Source location:** [`skills/wxo-validate-test-case/SKILL.md`](https://github.com/IBM/ibm-watsonx-orchestrate-adk/blob/main/skills/wxo-validate-test-case/SKILL.md) in the ADK repository.

## Overview

A unified test case has `tool_call` steps and `assert_llm` assertions in `goal_details`, and a dependency graph in `goals`. For the format, see [wxo-test-case-generation](https://developer.watson-orchestrate.ibm.com/bob_skills/wxo_test_case_generation.md#test-case-format).

The skill checks that the agent calls the declared tools in a valid order. It doesn't evaluate the `assert_llm` assertions.

**Best for:** Running regression or acceptance checks on existing test case files against a deployed agent.

## Key capabilities

* **Flexible input**: Accepts a single `.json` test case file or a directory of test case files.
* **Fixture checks before the live run**: Validates the `goals` graph and the structure of each test case before contacting the agent.
* **Directed conversation**: Starts a fresh conversation for each test with the `chat_with_agent` tool of the [watsonx Orchestrate ADK MCP server](https://developer.watson-orchestrate.ibm.com/mcp_server/wxOmcp_tools.md), and keeps one thread for chained tool steps.
* **Failure classification**: Classifies each failed run.
* **Remediation**: Offers to fix or regenerate test cases that are classified as bad.

## Prerequisites

* Python environment for your project that has the evaluation framework (`agentops`) installed. The skill runs a preflight check and stops if the import fails.
* The [watsonx Orchestrate ADK MCP server](https://developer.watson-orchestrate.ibm.com/mcp_server/wxOmcp_installation.md) configured in Bob and connected to the environment where your agent is imported. If the `chat_with_agent` tool isn't available, the skill stops and reports the configuration problem.

## How validation works

1. **Configure**: You provide the path to a test case file or directory. Results are returned in the response. To write result files, ask Bob for durable output. The default directory is `results/strict-eval-run`.
2. **Evaluate**: For each test, Bob starts a conversation with the test's `starting_sentence`. If the agent doesn't call the next expected tool, Bob sends a follow-up message in the same thread that asks the agent to call it with the expected arguments. Tool steps can complete in any order unless the graph makes one depend on another.
3. **Classify**: Bob classifies each failed test.
4. **Report**: Bob presents a run table, a failure classification table, and an aggregate summary.
5. **Remediate**: If any test is classified as `BAD_TEST_CASE`, Bob lists the affected files and offers to regenerate them with [wxo-test-case-generation](https://developer.watson-orchestrate.ibm.com/bob_skills/wxo_test_case_generation.md).

Because every argument uses `ignore` matching, validation doesn't treat a difference in argument values as a failure. A test case that uses another matching strategy is reported as unsupported, without a live run.

The conversation is limited to `2 * number of tool steps + 2` turns.

### Failure classes

| Class | Meaning |
| - | - |
| `AGENT_FAILURE` | The agent called an unexpected tool, skipped a dependency, called tools in an invalid order, or didn't make an expected call within the turn limit. |
| `EXTERNAL_FAILURE` | The run failed because of an SDK failure, timeout, unavailable agent, connection or authentication error, or malformed tool response. |
| `BAD_TEST_CASE / invalid_fixture` | The graph or structure of the test case is invalid. |
| `BAD_TEST_CASE / stale_data` | A success scenario needs an entity that doesn't exist at runtime. |
| `BAD_TEST_CASE / underspecified_story` | The story doesn't provide the information needed to drive the tool chain. |

An expected not-found response in an intentional negative test isn't a failure.

## Activation

In Bob, ask to validate a test case file or a directory:

```text theme={null}
Validate the test cases in ./evals/hr_agent.
```

## Next steps

To create new test cases, use the [wxo-test-case-generation](https://developer.watson-orchestrate.ibm.com/bob_skills/wxo_test_case_generation.md) skill.