Tool Performance Guide

Documentation Index

Fetch the complete documentation index at: https://developer.watson-orchestrate.ibm.com/llms.txt

Use this file to discover all available pages before exploring further.

​

Overview

​

Tool Types in wxO

​

Choosing the Right Tool Type

​

Tool Type Comparison

​

Understanding Tool Performance

​

The Granularity Principle

1. Tool Call Overhead: The cost of invoking a tool
   • Includes initialization, context setup, and result handling
   • Exists for every tool invocation
   • Relatively consistent per tool type
2. Execution Inside the Tool: The actual work being done
   • Runs fast once inside the tool
   • Varies based on the logic and operations
   • Where optimization efforts should focus

• Multiple small tool calls = Multiple overhead costs
• Single larger tool call = One overhead cost, fast internal execution
• Recommendation: Combine related operations into single tools when possible

​

Performance Characteristics by Tool Type

• Call overhead: Present but minimal
• Internal execution: Very fast for most operations
• Overall: Fast to very fast
• Best for: Deterministic logic, data processing, API calls

• Call overhead: Includes Langflow initialization (2+ seconds due to initialization)
• Internal execution: Depends on LLM operations
• Overall: Slower due to LLM inference (if using LLM)
• Best for: LLM reasoning, NLP tasks, document analysis

​

Python & Langflow Tool Performance

​

Shared Technical Constraints

​

Python Tools

• Minimize and batch external API calls
• Use efficient algorithms (O(n) vs O(n²))
• Choose performant libraries
• Implement external caching (Redis) for expensive operations

​

Overview

Key changes in Python toolkits

• New toolkit runtime: Overcomes individual tool processing overheads.
• Improved performance: Improves resource utilization and reduces cold start latency.

​

New toolkit runtime architecture

Architecture overview

• Each tool has its own requirements.txt with specific dependencies
• Process-level isolation: Each tool invocation creates a new process, albeit a lightweight one.
• Thread-safety: Tools need not be thread-safe.
• Performance overhead: Creates a lightweight process for each execution and a virtual environment for tools that are not yet loaded in the worker.
• Resource allocation: 2 vCPU and 2 GB RAM per replica (2 replicas provided)
• Cold-start delays: Significant latency from process creation and dependency loading.

• Multiple related tools packaged together in a single toolkit.
• Single shared requirements.txt for all tools in the toolkit.
• Thread-level isolation: Tools run in persistent worker threads.
• Preinstalled dependencies: All dependencies are loaded once at container startup.
• Thread-safety required: All tools must be thread-safe and reentrant.
• High performance: No process creation or dependency loading overhead.
• Resource allocation: 2 vCPU and 2 GiB RAM per replica (2 replicas provided).
• FastAPI/Gunicorn workers: 5 persistent workers per replica handle concurrent requests.

Deployment architecture

• Kubernetes deployment: 2 replicas
• Per-Replica resources:
  • 2 vCPU cores
  • 2 GiB memory
  • 5 FastAPI/Gunicorn workers per replica
• Load balancing: Automatic distribution across replicas and workers
• Health monitoring: Built-in health checks and automatic recovery

Benefits of toolkit approach

1. Higher performance:
   • No process creation overhead per execution.
   • No dynamic dependency loading.
   • Persistent workers eliminate cold start delays.
   • Thread-based execution is orders of magnitude faster than process-based.
2. Shared dependencies:
   • Single requirements.txt for all tools in the toolkit
   • Dependencies loaded once at container startup
   • Reduced memory footprint through shared libraries
3. Better resource utilization:
   • Multiple tools share container resources
   • Efficient thread-based concurrency
   • Lower per-request overhead
4. Improved scalability:
   • Each replica handles multiple concurrent requests through workers
   • Better throughput per unit of compute
5. Enhanced reliability:
   • Built-in health checks and automatic failover
   • Persistent workers reduce failure points
   • No dependency loading failures during execution
6. Cost efficiency:
   • More tools per unit of compute resource
   • Reduced infrastructure overhead
   • Better resource density

Thread safety requirements

• Multiple worker threads run tools concurrently
• No process-level isolation between tool invocations
• Shared memory space within the container

# ✅ CORRECT: Thread-safe tool (no shared mutable state)
@tool
async def thread_safe_tool(user_id: str) -> Dict:
    """Each invocation uses local variables only"""
    async with httpx.AsyncClient() as client:
        response = await client.get(f"https://api.example.com/users/{user_id}")
        return response.json()

# ❌ INCORRECT: Not thread-safe (shared mutable state)
_cache = {}  # Shared across all threads!

@tool
async def not_thread_safe(key: str) -> str:
    """Dangerous: modifying shared state without locks"""
    if key not in _cache:
        _cache[key] = await fetch_data(key)  # Race condition!
    return _cache[key]

​

Resource allocation and performance requirements

CPU computation limits

• Nonblocking execution in the event loop
• Fair resource sharing across concurrent requests
• Predictable response times
• Prevention of worker starvation

What counts as CPU time

• Data parsing and transformation logic
• JSON serialization/deserialization
• String manipulation and formatting
• Mathematical computations
• In-memory data filtering and sorting
• Object instantiation and manipulation

• Network I/O wait time (HTTP requests, database queries)
• File I/O wait time
• Sleep/delay operations
• External API call latency

Memory constraints

• Per-worker budget: ~400 MB (accounting for overhead).
• Tool memory usage: Cannot not exceed 100-200 MB per concurrent request.
• Memory leaks: Avoid memory leaks to prevent pod crashes.
• Throttling: Implement request throttling if memory usage approaches limits.

​

Python tool design principles

1. Tools as API wrappers

from ibm_watsonx_orchestrate.agent_builder.tools import tool
import httpx
from typing import Dict, List

@tool
async def get_customer_orders(customer_id: str) -> Dict:
    """
    Fetch customer orders from multiple services and format the response.
    
    This tool demonstrates the correct pattern:
    - Heavy lifting done by external APIs
    - Async I/O for non-blocking execution
    - Minimal CPU time for response formatting
    """
    async with httpx.AsyncClient() as client:
        # API call 1: Get customer profile (async, non-blocking)
        profile_response = await client.get(
            f"https://api.example.com/customers/{customer_id}"
        )
        profile = profile_response.json()
        
        # API call 2: Get order history (async, non-blocking)
        orders_response = await client.get(
            f"https://api.example.com/orders?customer_id={customer_id}"
        )
        orders = orders_response.json()
        
        # API call 3: Get loyalty points (async, non-blocking)
        loyalty_response = await client.get(
            f"https://api.example.com/loyalty/{customer_id}"
        )
        loyalty = loyalty_response.json()
        
        # API call 4: Get recommendations (async, non-blocking)
        recs_response = await client.get(
            f"https://api.example.com/recommendations/{customer_id}"
        )
        recommendations = recs_response.json()
        
        # CPU work: Format and combine responses (~10-20ms)
        result = {
            "customer_name": profile.get("name"),
            "email": profile.get("email"),
            "total_orders": len(orders.get("items", [])),
            "recent_orders": orders.get("items", [])[:5],
            "loyalty_points": loyalty.get("points", 0),
            "tier": loyalty.get("tier", "standard"),
            "recommended_products": [
                {
                    "id": rec["product_id"],
                    "name": rec["product_name"],
                    "score": rec["relevance_score"]
                }
                for rec in recommendations.get("items", [])[:3]
            ]
        }
        
        return result

2. Mandatory async I/O

# ✅ CORRECT: Async I/O
@tool
async def fetch_data(url: str) -> dict:
    async with httpx.AsyncClient() as client:
        response = await client.get(url)
        return response.json()

# ❌ INCORRECT: Blocking I/O
@tool
def fetch_data_blocking(url: str) -> dict:
    import requests
    response = requests.get(url)  # Blocks the event loop!
    return response.json()

• Allows the worker to handle other requests while it waits for the I/O
• Enables high concurrency with limited resources
• Prevents worker starvation
• Maximizes throughput

3. Response formatting guidelines

# ✅ CORRECT: Minimal formatting
@tool
async def process_api_response(data: dict) -> dict:
    """Simple field extraction and renaming"""
    return {
        "id": data.get("customer_id"),
        "name": data.get("full_name"),
        "status": data.get("account_status", "active")
    }

# ⚠️ CAUTION: More complex but still acceptable
@tool
async def aggregate_metrics(data: List[dict]) -> dict:
    """Light aggregation - keep under 50ms"""
    total = sum(item.get("value", 0) for item in data)
    avg = total / len(data) if data else 0
    return {
        "total": total,
        "average": avg,
        "count": len(data)
    }

# ❌ INCORRECT: Heavy computation
@tool
async def complex_analysis(data: List[dict]) -> dict:
    """This will exceed 50ms CPU time"""
    # Heavy statistical analysis
    # Machine learning inference
    # Complex data transformations
    # These should be done by external APIs!
    pass

​

Performance analysis and capacity planning

Example scenario: high-throughput tool

• CPU time per request: 50 ms (formatting/parsing)
• API call latency: 15 seconds each (4 calls in parallel via async)
• Total wall-clock time: ~15 seconds (parallel execution)

• 2 Kubernetes replicas
• 2 vCPU per replica
• 5 workers per replica
• Total workers: 10 (2 replicas × 5 workers)

Throughput calculation

• Total vCPUs: 4 (2 replicas × 2 vCPU)
• CPU time per request: 50 ms = 0.05 seconds
• Typical CPU-based throughput: 50 requests/second

​

When to use platform-hosted Python toolkits

✅ Suitable use cases

1. API orchestration tools
   • Calling 2-5 external APIs and combining results
   • Response formatting and field mapping
   • Simple data aggregation from multiple sources
2. Data transformation tools
   • JSON/XML format conversions
   • Field extraction and renaming
   • Simple filtering and sorting
   • Data validation and sanitization
3. Integration wrappers
   • Wrapping REST APIs with simplified interfaces
   • Authentication token management
   • Request/response normalization
4. Lightweight processing
   • Text formatting and templating
   • Date/time conversions
   • Unit conversions
   • Simple calculations (< 50 ms CPU)

Requirements checklist

• CPU computation ≤ 50 ms per request
• All I/O operations use async/await
• Memory usage is less than 200 MB per request
• Heavy processing is delegated to external APIs
• Response time is acceptable at 5-30 seconds
• The tool is stateless or uses platform context management
• No special runtime dependencies (such as custom C libraries)

​

When to use self-hosted solutions

❌ Not suitable for platform toolkits

1. CPU-intensive operations
   • Complex statistical analysis
   • Cryptographic operations
   • Large-scale data transformations
2. High memory requirements
   • Loading large models (> 500 MB)
   • Processing large datasets in memory
   • Caching extensive reference data
   • In-memory databases
3. Special runtime requirements
   • Custom system libraries
   • Specific Python versions
   • Native code dependencies
4. Long-running operations
   • Batch processing jobs
   • Report generation (> 60 seconds)
   • Data migration tasks
   • Background processing
5. Stateful operations
   • Maintaining persistent connections
   • Session management
   • Transaction coordination
   • Workflow orchestration

Self-hosting options

• Best for: Custom tool collections with complex logic
• Deployment: Your infrastructure
• Protocol: Model Context Protocol over HTTP/SSE
• Benefits: Full control, custom resources, flexible scaling

• Best for: Existing REST APIs
• Deployment: Your infrastructure or third-party services
• Protocol: Standard HTTP/REST
• Benefits: Use existing APIs, no code changes needed

• Platform toolkits for orchestration and formatting
• Self-hosted services for heavy computation
• Example: Platform tool calls your ML inference API

​

Best practices

1. Async I/O patterns

import httpx
import asyncio
from typing import List, Dict

@tool
async def parallel_api_calls(ids: List[str]) -> List[Dict]:
    """Efficiently fetch data for multiple IDs in parallel"""
    async with httpx.AsyncClient() as client:
        # Create tasks for parallel execution
        tasks = [
            client.get(f"https://api.example.com/items/{id}")
            for id in ids
        ]
        # Wait for all to complete
        responses = await asyncio.gather(*tasks)
        # Format results (minimal CPU time)
        return [r.json() for r in responses]

2. Error handling

@tool
async def robust_api_call(endpoint: str) -> Dict:
    """Proper error handling for external API calls"""
    try:
        async with httpx.AsyncClient(timeout=30.0) as client:
            response = await client.get(endpoint)
            response.raise_for_status()
            return response.json()
    except httpx.TimeoutException:
        return {"error": "API timeout", "status": "timeout"}
    except httpx.HTTPStatusError as e:
        return {"error": f"API error: {e.response.status_code}", "status": "error"}
    except Exception as e:
        return {"error": str(e), "status": "error"}

3. Connection pooling

# Module-level client for connection reuse
_http_client = None

async def get_http_client():
    """Reuse HTTP client across requests"""
    global _http_client
    if _http_client is None:
        _http_client = httpx.AsyncClient(
            timeout=30.0,
            limits=httpx.Limits(max_keepalive_connections=20)
        )
    return _http_client

@tool
async def efficient_api_call(url: str) -> Dict:
    """Use connection pooling for better performance"""
    client = await get_http_client()
    response = await client.get(url)
    return response.json()

4. Memory management

@tool
async def memory_efficient_processing(data_url: str) -> Dict:
    """Stream large responses instead of loading into memory"""
    async with httpx.AsyncClient() as client:
        async with client.stream('GET', data_url) as response:
            # Process in chunks
            total_size = 0
            async for chunk in response.aiter_bytes(chunk_size=8192):
                total_size += len(chunk)
                # Process chunk without loading entire response
            
            return {"processed_bytes": total_size}

5. Context management

from ibm_watsonx_orchestrate.agent_builder.tools import tool
from ibm_watsonx_orchestrate.utils import update_context
from ibm_watsonx_orchestrate.run.context import AgentRun

@tool
async def stateful_tool(user_input: str, context: AgentRun) -> str:
    """Use platform context for state management"""
    # Read previous state
    previous_step = context.request_context.get('workflow_step', 'start')
    
    # Perform operation
    result = await process_step(user_input, previous_step)
    
    # Update context for next tool
    update_context('workflow_step', result['next_step'])
    update_context('last_result', result['data'])
    
    return result['message']

​

Migration from individual tools to toolkits

Migration checklist

1. Group-related tools:
   • Identify tools that share similar dependencies
   • Group tools by functional domain or use case
   • Aim for 10-15 tools per toolkit for optimal resource utilization
2. Consolidate dependencies:
   • Merge all requirements.txt files into a single file
   • Resolve version conflicts between tools
   • Remove duplicate dependencies
   • Test compatibility of consolidated dependencies
3. Ensure thread safety:
   • Critical: Review all tools for thread-safety issues
   • Remove or protect a shared mutable state
   • Use thread-safe data structures (queue.Queue, threading.Lock)
   • Avoid global variables that are modified during execution
   • Test tools under concurrent load
4. Review CPU usage:
   • Profile each tool to help ensure CPU time < 50 ms
   • Identify and optimize CPU-intensive operations
   • Consider moving heavy computation to external APIs
5. Convert to async:
   • Replace all blocking I/O with async/await
   • Use async-compatible libraries (httpx, aiohttp, asyncpg)
   • Ensure all I/O operations are nonblocking
6. Test concurrency:
   • Load test with multiple concurrent requests
   • Verify no race conditions or deadlocks
   • Monitor memory usage under load
   • Validate thread-safety under stress
7. Package as toolkit:
   • Follow toolkit packaging format
   • Create single requirements.txt
   • Document thread-safety guarantees
   • Include toolkit metadata

​

Summary

• Eliminates cold start delays through persistent workers
• Maximizes throughput through async I/O and efficient resource sharing
• Ensures predictability through strict CPU time limits

​

Langflow Tools

• Use for longer-running operations where initialization overhead is proportionally smaller
• Minimize and combine LLM calls into single prompts (when using LLMs)
• Use concise, focused prompts with minimal context (when using LLMs)
• Choose smaller models for simple tasks, larger for complex reasoning (when using LLMs)
• Cache expensive operation results externally (Redis) for repeated queries

​

API & MCP Tool Performance

​

API Tools (OpenAPI-based)

• Synchronous calls: 2 minutes (same as Python/Langflow/MCP)
• Asynchronous calls: None (via OpenAPI callback syntax)

​

MCP Tools (Model Context Protocol)

​

Summary

• wxO Flow: First choice for most scenarios - supports connections, security, custom code, no timeout limit
• Python Tools: For custom libraries not supported in wxO Flow code blocks
• Langflow Tools: For Langflow-specific AI components
• API Tools: For OpenAPI-based external service integrations - performance depends on external service
• MCP Tools: For Model Context Protocol capabilities - performance varies by implementation
• Tool call overhead exists: Combine operations when possible
• 2-minute timeout: Applies to synchrous tools such as Python, Langflow, OpenAPI sync and MCP tools

• Main Performance Guide
• Agent Performance Guide
• Flow Performance Guide