> ## Documentation Index > Fetch the complete documentation index at: https://developer.watson-orchestrate.ibm.com/llms.txt > Use this file to discover all available pages before exploring further. # Tutorial: Automating IT Operations with Bob and watsonx Orchestrate IT Operations teams handle many repetitive, low-complexity incidents across multiple systems. Typical requests include VPN connectivity issues, application access problems, and password-related incidents. These requests often follow well-defined resolution procedures, but support teams still use significant time classifying incidents, creating tickets, identifying the correct runbooks, and documenting resolutions. IBM Bob and watsonx Orchestrate can help automate this process. In this tutorial, you use Bob to transform a Business Requirements Definition (BRD) into a Standard Operating Procedure (SOP). You can use that specification to build an incident triage solution in watsonx Orchestrate. You create 5 AI agents, an agent workflow, Python tools, and a knowledge base. You then test, evaluate, document, and deploy the solution. You start directly from watsonx Orchestrate by launching Bob. Bob automatically prepares the development environment, installs the watsonx Orchestrate ADK components, configures the required MCP servers, and loads specialized skills for solution architecture, SOP creation, agent development, testing, evaluation, and documentation. Using this specification-driven approach, you can move from business requirements to a working agent solution with clear traceability between requirements, implementation, and validation. ## Architecture of the incident triage solution The incident triage solution automates the handling of repetitive, low-complexity IT incidents. The solution uses a coordinated set of AI agents, an agent workflow, Python tools, and a knowledge base running on watsonx Orchestrate. Each agent is responsible for a specific task in the incident lifecycle, while the workflow manages the sequence of activities from incident submission to resolution. This architecture helps IT Operations teams reduce manual effort, improve response times, and deliver consistent incident handling across systems. By combining AI agents, business rules, and operational knowledge, the solution can resolve common incidents with minimal human intervention. Bob creates the following components: * `incident_triage_orchestration`: Workflow to manage the incident lifecycle from submission to resolution. * Five specialist agents: * `incident_triage_coordinator`: Handles user requests and starts the workflow. * `first_responder`: Classifies incidents and creates support tickets. * `runbook_specialist`: Retrieves root cause and resolution guidance from the knowledge base. * `remediation_engineer`: Executes remediation actions. * `summary_specialist`: Generates the final resolution summary. * `incident_runbooks`: Knowledge base containing sample runbooks for VPN connectivity, SCADA application, and billing access incidents. Before production deployment, replace these samples with your organization's runbooks. * Two Python tools: * `create_support_ticket`: Creates a support ticket and returns a ticket URL. * `execute_remediation_action`: Simulates a remediation action and returns the execution status and confirmation details. ### Solution workflow The user submits a free-text incident request to the `incident_triage_coordinator` agent. This agent starts the `incident_triage_orchestration` workflow. The workflow sends the request to the `first_responder` agent, which classifies the incident and creates a support ticket. The workflow then passes the request to the `runbook_specialist` agent, which retrieves the relevant root cause and resolution guidance from the `incident_runbooks` knowledge base. Next, the `remediation_engineer` agent performs the required remediation action. Finally, the `summary_specialist` agent generates a clear resolution summary in plain language. The large language model (LLM) helps the agents understand the incident request and available context. It supports incident classification, runbook-based decision making, action selection, tool usage, and response generation. The workflow, Python tools, and knowledge base provide the execution logic and business data that is required to complete the incident resolution process. ## Prerequisites Before you begin, make sure that you have the following prerequisites: * IBM Bob installed. [Sign up for access to IBM Bob](https://www.ibm.com/products/ibm-bob). * A watsonx Orchestrate account. [Sign up for a watsonx Orchestrate trial account](https://www.ibm.com/products/watsonx-orchestrate). * watsonx Orchestrate Developer Edition installed. Follow the [watsonx Orchestrate Developer Edition installation guide](../developer_edition/wxOde_setup). This tutorial was tested with Agent Development Kit (ADK) 2.15. ## Step 1. Initialize the Bob workspace In this step, you use Launch Bob in watsonx Orchestrate to create a ready-to-use Bob workspace. Bob installs Python if it is not already available, installs or updates the watsonx Orchestrate ADK, connects to your watsonx Orchestrate environment, and configures two MCP servers. One MCP server manages the watsonx Orchestrate environment, and the other provides documentation search. Bob also loads watsonx Orchestrate skills that support the agent development lifecycle. In this tutorial, you use: * sop-builder: To convert the BRD into a Standard Operating Procedure (SOP) * wxo-builder: To create and test agents, tools, workflows, and knowledge bases * agent-instructions-evaluator: To evaluate agent instructions and behavior * wxo-analyzer: To analyze and document the completed solution Follow these steps to launch Bob: Log in to your watsonx Orchestrate account. Click **Create agent**, then click **Launch Bob**. Click **Open IBM Bob** when prompted. Bob detects that the watsonx Orchestrate ADK extension is not installed and prompts you to install it. Click **Install Extension** and **Open URI**. Bob prompts you to select a workspace folder. Click **Open Folder** and create a new folder named `incident-triage-wxo-agents`. This folder is used as your project workspace. After you open the workspace folder, Bob automatically continues the setup process. Bob checks your local Python environment. If `uv` is not installed, Bob prompts you to install it. Click Install. Bob installs `uv` and creates the required Python 3.12 environment. Bob installs or updates the watsonx Orchestrate ADK, configures the watsonx Orchestrate MCP servers, connects to your watsonx Orchestrate environment, and loads the watsonx Orchestrate skills that are used in this tutorial. After the setup is complete, the agents in your watsonx Orchestrate environment appear in the right-side panel in Bob. Wait for the success message to appear, then press **Enter** to load the watsonx Orchestrate skills. Bob uses the watsonx Orchestrate ADK MCP server to retrieve the latest watsonx Orchestrate skills. When prompted, click **Approve for Task**. Click the **Explorer** tab in the left navigation pane and review the workspace structure that is created by Bob. The `.bob` folder contains the watsonx Orchestrate skills (`sop-builder`, `wxo-builder`, `agent-instructions-evaluator`, and `wxo-analyzer`) and the MCP server configurations (`watsonx-orchestrate-adk` and `watsonx-orchestrate-adk-docs`). Bob also creates the standard watsonx Orchestrate project folders, including `agents`, `connections`, `knowledge-bases`, `models`, `toolkits`, and `tools`, along with the Python virtual environment (`venv`) and supporting project files. Open `.bob/mcp.json` and confirm that `watsonx-orchestrate-adk` and `watsonx-orchestrate-adk-docs` are listed. Then, open the `.bob/skills` folder and confirm that the required skills, including `sop-builder`, `wxo-builder`, `agent-instructions-evaluator`, and `wxo-analyzer`, are available. ## Step 2. Generate a standard operating procedure from the BRD In this step, you use Bob to convert the incident triage Business Requirements Definition (BRD) into a Standard Operating Procedure (SOP). The SOP provides a clear and reviewable specification for the solution. It defines the process flow, user roles, data flow, procedure steps, decision rules, exception handling, test scenarios, acceptance criteria, and integration requirements. This approach follows specification-driven development. The SOP serves as the implementation blueprint that Bob uses to design and build the solution. Download the `incident-triage-brd.md` file and save it in a new folder named `docs` inside your workspace. Click **New Task**. Because you are creating a specification document, switch to **Plan** mode. Enter the following prompt to generate the SOP: ``` Create SOP for this BRD @docs/incident-triage-brd.md ``` Bob automatically activates the `sop-builder` skill because the request matches the skill's purpose. Bob analyzes the prompt, selects the appropriate skill from the watsonx Orchestrate skills available in the workspace, and uses it to create the SOP. Bob also creates a subagent to review the workspace content and gather the required context. Bob generates the SOP and creates the sections that are required to implement the solution. These sections include the executive summary, business process flow diagram, business context, procedure overview, data requirements, agent instructions, procedure steps, decision logic, exception handling, test scenarios, and acceptance criteria. Review the SOP to confirm that it matches the business requirements. You can also switch to **Ask** mode and ask Bob questions about the SOP or request updates. ## Step 3. Create AI agents and the incident triage workflow In this step, you use Bob to implement the solution defined in the SOP. Bob creates the agents, agent workflow, knowledge base, and Python tools required for the incident triage solution. Click **Start New Task** to begin a new task. Switch to **Agent** mode so that Bob can create the required files, code, and configuration artifacts. Enter the following prompt to create the solution components: ``` Build the agents and agentic workflow on watsonx Orchestrate based on this SOP @docs/incident-triage-sop.md and BRD @docs/incident-triage-brd.md ``` Bob automatically activates the `wxo-builder` skill because the request matches its purpose. Bob reads the SOP and BRD documents and uses them as the implementation blueprint for the agents, workflow, tools, and knowledge base. Bob creates a task list that outlines the work that is required to build the solution. Bob creates the agents, Python tools, agent workflow, knowledge base, and unit tests. Bob then prompts you to import the solution into watsonx Orchestrate. Click **Approve for Task** to allow Bob to run the import script. Bob runs the unit tests, identifies issues, and fixes them automatically. Bob uses the `wxo-builder` skill to review errors, update the implementation, and validate the fixes. Bob runs smoke tests on the deployed agents by using the Chat with Agent MCP tool that is provided by the watsonx Orchestrate MCP server. When prompted, click **Approve for Task**. Bob tests multiple incident scenarios and generates a test report that summarizes the test results. After all tests pass, Bob verifies that the agents, workflow, tools, and knowledge base are created and deployed successfully. Bob then marks all implementation tasks as complete. ## Step 4. Evaluate and analyze the solution In this step, you evaluate the quality of the deployed agent instructions and assess whether they are ready for production use. You use the `agent-instructions-evaluator` skill to score each agent, identify issues, and receive improvement recommendations. Then, you use the `wxo-analyzer` skill to analyze the solution and generate documentation. The analysis includes solution architecture, agent interactions, tools, workflows, and supporting components. Click **Start New Task** to begin a new task. Enter the following prompt: ``` Evaluate all agents in @agents/ ``` Bob activates the `agent-instructions-evaluator` skill and evaluates the agent instructions. The evaluation scores each agent across multiple quality categories and provides recommendations for improvement. Bob generates an agent evaluation report that includes agent scores, cross-agent observations, system-level findings, runtime risks, and prioritized recommendations. Use this report to identify instruction gaps, workflow issues, and improvement opportunities before production testing. If needed, ask Bob to implement the recommended improvements from the evaluation report. Click **Start New Task**, then enter the following prompt: ``` Analyze this watsonx Orchestrate project and create a report. ``` Bob activates the `wxo-analyzer` skill and analyzes the project. Bob generates three reports that document the solution architecture, agent implementation, and supporting components. The reports include Mermaid diagrams, component inventories, relationships, configuration details, dependencies, and implementation findings. These reports make the solution easier to review, troubleshoot, and maintain. The two Bob skills serve different purposes. The `agent-instructions-evaluator` skill evaluates agent instructions, agent interactions, and production readiness. It identifies issues and recommends improvements. The `wxo-analyzer` skill documents the solution architecture, agents, tools, workflows, knowledge bases, dependencies, and implementation details, making the solution easier to understand and manage. ## Step 5. Verify the solution in watsonx Orchestrate In this step, you verify the incident triage solution in watsonx Orchestrate. You test the solution with a sample incident, review the created agents, tools, and knowledge base, and examine the workflow that is created by Bob. Log in to watsonx Orchestrate and open the `incident_triage_coordinator` agent. Select the **Network Outage** sample request and submit it to the agent. Review the response and verify that the incident is processed correctly. Click **Manage Agents** and review the solution components that are created by Bob, including the five agents, Python tools, and the knowledge base. Open the `incident_triage_coordinator` agent and review its instructions, configuration, and behavior. Then, open the **Tools** tab to view the tools that are assigned to the agent. Click **Open Flow Builder** to view the incident triage workflow created by Bob. Explore the workflow steps and agent interactions. ## Step 6. Add stand-alone tools for existing ticket requests When you verified the solution, you saw that the `incident_triage_orchestration` workflow follows a predefined sequence based on the BRD. The workflow calls the `first_responder`, `runbook_specialist`, `remediation_engineer`, and `summary_specialist` agents in a specific order. This approach works well for new incidents that require the complete triage and resolution process. The business has now identified extra support scenarios for existing tickets. These requests do not require the full workflow. In this step, you ask Bob to create two stand-alone Python tools that can query and update existing support tickets. You will then add these tools directly to the `incident_triage_coordinator` agent. The agent can use natural language reasoning to determine whether it should start the incident triage workflow, retrieve information about an existing ticket, or update an existing ticket. This example shows how watsonx Orchestrate combines structured workflows with intelligent decision making. The workflow handles multi-step incident resolution processes, while the agent uses reasoning to select the appropriate stand-alone tool for simpler requests. This approach provides flexibility while maintaining control and consistency for business processes. Click **Start New Task** to begin a new task. Make sure that Bob is in **Agent** mode. Because this is a small enhancement, you can ask Bob to implement the change directly without creating a separate plan. Enter the following prompt: ``` Create two standalone Python tools to query and update existing support tickets. Update the incident_triage_coordinator agent YAML to use these tools for existing tickets or invoke the incident_triage_orchestration workflow for new incidents. ``` Bob creates the `query_support_ticket` and `update_support_ticket` Python tools. Bob also updates the `incident_triage_coordinator` agent so it can route requests based on user intent. The agent can query an existing ticket, update an existing ticket, or start the `incident_triage_orchestration` workflow for a new incident. Bob updates the import and deletion scripts, adds starter prompts, deploys the updated agent and tools, and validates all three request paths: * Query an existing support ticket * Update an existing support ticket * Start the `incident_triage_orchestration` workflow for a new incident Log in to watsonx Orchestrate and open the `incident_triage_coordinator` agent. Submit a request such as: ``` What is the current status of ticket TKT-A1B2C3D4? ``` Review the agent response. Notice that the agent recognizes the request as a ticket status inquiry and automatically calls the `query_support_ticket` tool instead of starting the `incident_triage_orchestration` workflow. ## Summary and next steps In this tutorial, you used Bob to build an incident triage solution from a Business Requirements Definition (BRD). You converted the BRD into a Standard Operating Procedure (SOP) and used that specification to create, test, evaluate, and document AI agents, an agent workflow, Python tools, and a knowledge base in watsonx Orchestrate. Bob served as an AI-powered development assistant throughout the process. Using watsonx Orchestrate skills and MCP servers, Bob converted business requirements into implementation artifacts, generated the required components, performed unit and smoke testing, evaluated agent quality, and created solution documentation. You also enhanced the solution by adding stand-alone tools for existing ticket management. This demonstrated how watsonx Orchestrate can combine structured workflows with agent reasoning. The incident workflow handles multi-step incident resolution, while the coordinator agent uses natural language understanding to select the appropriate workflow or tool based on the user's request. Bob completed the following tasks in this tutorial: | Task | Description | | ------------------------ | ---------------------------------------------------------------------------- | | Workspace initialization | Installed ADK, configured MCP servers, and loaded watsonx Orchestrate skills | | SOP generation | Converted the BRD into a structured Standard Operating Procedure | | Solution build | Created 5 agents, 1 workflow, 2 Python tools, and 1 knowledge base | | Testing | Ran unit tests and smoke tests, and fixed issues automatically | | Evaluation | Scored agent instructions and provided improvement recommendations | | Documentation | Generated architecture and implementation reports | | Enhancement | Added stand-alone tools and updated agent routing logic | ### Next steps * Review the [ADK extension overview](./overview) to learn more about Bob workspace capabilities. * Explore the [watsonx Orchestrate ADK MCP server](../mcp_server/wxOmcp_installation) for extra management tools. * Learn how to [configure access to remote environments](../environment/initiate_environment) for production deployment.