Skip to main content
New in 2.20.0
The wxo-validate-test-case skill validates unified test case JSON files against a live watsonx Orchestrate agent. It directs the agent through the tool chain that the test case declares, and classifies any failures. Source location: skills/wxo-validate-test-case/SKILL.md in the ADK repository.

​
Overview

A unified test case has tool_call steps and assert_llm assertions in goal_details, and a dependency graph in goals. For the format, see wxo-test-case-generation. The skill checks that the agent calls the declared tools in a valid order. It doesn’t evaluate the assert_llm assertions. Best for: Running regression or acceptance checks on existing test case files against a deployed agent.

​
Key capabilities

  • Flexible input: Accepts a single .json test case file or a directory of test case files.
  • Fixture checks before the live run: Validates the goals graph and the structure of each test case before contacting the agent.
  • Directed conversation: Starts a fresh conversation for each test with the chat_with_agent tool of the watsonx Orchestrate ADK MCP server, and keeps one thread for chained tool steps.
  • Failure classification: Classifies each failed run.
  • Remediation: Offers to fix or regenerate test cases that are classified as bad.

​
Prerequisites

  • Python environment for your project that has the evaluation framework (agentops) installed. The skill runs a preflight check and stops if the import fails.
  • The watsonx Orchestrate ADK MCP server configured in Bob and connected to the environment where your agent is imported. If the chat_with_agent tool isn’t available, the skill stops and reports the configuration problem.

​
How validation works

  1. Configure: You provide the path to a test case file or directory. Results are returned in the response. To write result files, ask Bob for durable output. The default directory is results/strict-eval-run.
  2. Evaluate: For each test, Bob starts a conversation with the test’s starting_sentence. If the agent doesn’t call the next expected tool, Bob sends a follow-up message in the same thread that asks the agent to call it with the expected arguments. Tool steps can complete in any order unless the graph makes one depend on another.
  3. Classify: Bob classifies each failed test.
  4. Report: Bob presents a run table, a failure classification table, and an aggregate summary.
  5. Remediate: If any test is classified as BAD_TEST_CASE, Bob lists the affected files and offers to regenerate them with wxo-test-case-generation.
Because every argument uses ignore matching, validation doesn’t treat a difference in argument values as a failure. A test case that uses another matching strategy is reported as unsupported, without a live run. The conversation is limited to 2 * number of tool steps + 2 turns.

​
Failure classes

An expected not-found response in an intentional negative test isn’t a failure.

​
Activation

In Bob, ask to validate a test case file or a directory:

​
Next steps

To create new test cases, use the wxo-test-case-generation skill.