wxo-test-case-generation skill

​

Overview

1. Agent capabilities
2. Scenario families
3. Proposed test cases
4. Clarification of details that define the test, if needed
5. A generated test case

​

Key capabilities

• Scenario discovery: Groups the agent’s tools into scenario families and proposes concrete test cases for the ones you choose.
• Minimal clarification: Asks only about inputs that define the test.
• Unified test case output: Generates a single JSON file that contains tool_call steps, a dependency graph, and assert_llm assertions.
• Validation against a live agent: Runs a format check, then validates the test by directing the agent through the declared tool chain with the chat_with_agent tool of the watsonx Orchestrate ADK MCP server.
• Post-validation review: Runs a story review and an assertion review on the observed conversation, and revises the story and assert_llm entries.
• Upload: Uploads approved test cases with the upload_test_case tool.
• Summary: Writes a summary.md file that lists the scenarios you chose, the test cases created, and where each file is saved.

​

Prerequisites

• Python environment for your project that has the evaluation framework (agentops) installed. The skill runs a preflight check and stops if the import fails.
• The watsonx Orchestrate ADK MCP server configured in Bob, connected to the environment where your agent is imported. The skill uses the MCP server for validation and upload.
• The agent definition file and the tool definitions in your current directory. If Bob can’t find them, it asks you for the paths. Without them, test case generation might be less accurate.

​

Test case format

{
  "agent": "hr_agent",
  "goals": {
    "lookup_employee": []
  },
  "goal_details": [
    {
      "name": "lookup_employee",
      "tool_name": "lookup_employee",
      "type": "tool_call",
      "args": {"employee_id": "E1001"},
      "arg_matching": {"employee_id": "ignore"}
    },
    {
      "type": "assert_llm",
      "kind": "observed",
      "scope": "conversation",
      "check": "Agent completed the task requested in the story and told the user the outcome."
    }
  ],
  "story": "",
  "starting_sentence": ""
}

​

Tool steps

• Each tool_call step keeps its expected arguments in args.
• Every argument has an arg_matching value of ignore. The expected values guide the instructions that Bob sends to the agent, but validation doesn’t compare them against the values that the agent produces.
• Name repeated calls to the same tool tool_name_1, tool_name_2, and so on, and keep the real tool_name in each step.

​

Assertions

​

Activation

Generate test cases for my hr_agent.

Generate a vacation-related test for hr_agent.

​

What happens after a test case is generated

1. Bob validates the test case against the live agent and reports the run status and the number of matched tool steps.
2. Bob reviews the conversation and revises the story and assertions.
3. You review the test case and request changes if needed.
4. When you approve the test case, Bob asks where to save it. The default path is {output_folder}/{scenario_id}/test-{proposal_id}.json.
5. If you ask for an upload, Bob uploads the saved file with the upload_test_case MCP tool. Bob reports success only after the MCP server confirms it.

​

Next steps