> ## Documentation Index > Fetch the complete documentation index at: https://developer.watson-orchestrate.ibm.com/llms.txt > Use this file to discover all available pages before exploring further. # Knowledge and document processing considerations The following table covers the most common knowledge and document processing anti-patterns that affect agent response quality and accuracy. ##### ⭐️ Most common anti-patterns | Anti-pattern | Problem | Impact | Fix | | ---------------------------------------- | -------------------- | ------------------ | -------------------------------------- | | Assumed trust in the knowledge source | Assume truth | Amplified errors | Verification | | ⭐️ 'RAG will fix disorganized knowledge' | Poor structure | Bad outputs | Fix knowledge | | Knowledge retrieval token bloat | Over-fetching | Large token waste | Top-N retrieval and organize knowledge | | ⭐️ One-size-fits-all chunking | Fixed chunks | Broken semantics | Content-aware | | Using RAG for whole-document operations | Wrong tool | Structure loss | Use DB/API | | Ignoring file format implications | Same processing | Lost structure | Format-aware | | Mixing extraction with processing | Overloaded extractor | Errors | Separate steps | | Lazy field definitions | Vague schema | Fuzzy field errors | Explicit fields | #### What it is * Assuming whatever RAG retrieves is inherently correct * Treating vector stores, CMS, Confluence, SharePoint as curated sources of truth * No validation of knowledge quality before or after retrieval #### Why it fails * Enterprise content is rarely clean (outdated policies, conflicting versions, stale information) * RAG retrieves semantically similar text, not verified truth (similar ≠ correct) * Agents hallucinate within retrieved content when chunks have gaps * RAG can't distinguish policy versioning unless explicitly encoded * Agents can't detect contradictions between conflicting chunks * All content treated equally (high-risk policy = informal wiki page) #### What to do instead: Knowledge quality is a first-class concern * **Knowledge verification pipelines:** Validate content for correctness, consistency, recency, conflicts, versioning before indexing * **Metadata-driven retrieval:** Add version, effective date, document type, risk classification, authoritative owner, region/jurisdiction * **Policy-first structuring:** Extract canonical definitions, rules, conditions, exceptions, thresholds, regulatory constraints; expose through structured APIs * **Retrieval confidence + cross-checking:** Let agents query authoritative sources, compare results, escalate contradictions, ask follow-up questions * **Human-in-the-loop for high-risk domains:** Require review for compliance/legal/medical content; provide "verified content only" modes #### What it is * Assuming semantic search can magically infer structure, correctness, and intent from messy content * Believing RAG compensates for poor knowledge architecture * No cleanup of thousands of pages across Confluence, PDFs, SharePoint, wikis, legacy docs #### Why it fails * RAG amplifies disorganization, doesn't correct it * RAG returns whatever is "closest," not whatever is "true" * Disorganized sources produce conflicting chunks that agents merge incorrectly * Semantic retrieval has no notion of versioning or authority * Large, messy spaces generate retrieval flooding (noise vs. signal) * RAG cannot infer structure that doesn't exist * Garbage in → amplified garbage out (with confident-sounding errors) #### What to do instead: Prepare knowledge for agents, not humans * **Identify knowledge types:** Definitions, rules/constraints, procedures, exceptions, facts, disclosures, reference material, narrative context (RAG treats these identically; agents do not) * **Restructure into agent-friendly formats:** Long PDFs → rule tables/schemas; paragraphs → JSON rules; buried instructions → deterministic workflow steps; mixed definitions → canonical glossary tool; conflicting versions → versioned hierarchy * **Use RAG only for right categories:** Excellent for concept retrieval, domain context, semantic interpretation, large knowledge spaces; terrible for factual truth, procedural rules, policy enforcement, compliance, exactness * **Make knowledge quality a responsibility:** Introduce metadata, ownership, validation, versioning, proper indexing, lifecycle governance #### What it is * Returning excessive knowledge passages that get sent to the LLM on every conversation turn * Knowledge passages, once retrieved, are included in context for every subsequent turn * Teams return 10-15 passages when 2-3 would suffice ("just in case" over-retrieval) #### Why it fails * **Compounding token costs:** 20K tokens × 5 turns = 100K tokens of redundant context * **Context exhaustion and attention dilution:** Large retrievals consume context budget and weaken focus * **Per-turn cost multiplication:** Every token retrieved is sent repeatedly throughout conversation #### What to do instead: Precision retrieval with token awareness * **Precision over recall:** Return top 3-5 passages, not 10-15; use higher similarity thresholds * **Progressive retrieval:** Start with minimal, high-confidence passages; retrieve more only when needed * **Monitor utilization:** Track tokens per turn, retrieval utilization, redundancy rate, cost per conversation * **Principle:** Knowledge retrieval is not a one-time cost—it's a per-turn cost. Optimize for precision, not coverage. #### What it is * Using default chunk sizes (512 or 1024 tokens) without understanding document content or use case * Treating chunking as one-time configuration rather than content-aware strategy * Assuming single chunk size works for FAQs, legal documents, technical manuals, and product catalogs #### Why it fails * **Different content has different natural boundaries:** FAQs have Q\&A pairs, legal docs have clauses, manuals have procedures * **Default sizes are arbitrary:** Not chosen for your content—may split critical information or group irrelevant content * **Size mismatches create problems:** Too large = "needle in haystack"; too small = broken answers requiring synthesis * **Content structure ignored:** Tables, lists, code blocks, hierarchical sections need different strategies than prose #### What to do instead: Content-aware chunking strategy * **Match strategy to content type:** FAQs (100-300 tokens per Q\&A pair), Legal/policy (300-800 tokens per clause), Technical manuals (200-500 tokens per procedure), Product catalogs (50-200 tokens per product) * **Preserve semantic boundaries:** Don't split mid-paragraph or mid-concept * **Use overlapping chunks:** Provide context continuity at boundaries (10-20% overlap) * **Test with real queries:** Measure retrieval quality, not just similarity scores * **Monitor multi-chunk needs:** Track how often agents need multiple chunks to answer questions #### What it is * Using agent knowledge for questions requiring access to entire document or corpus * Expecting RAG to handle counting, aggregation, ordering, or document structure operations #### Why it fails * RAG retrieves top matching chunks (5-10), not complete documents * Counting, aggregation, ordering require full dataset access * Document structure (rows, pages, sections) is lost in chunking * File type and metadata aren't preserved in vector embeddings * Agent tries to retrieve rows matching query words rather than operating on original data representation #### What to do instead: Use databases and APIs, not RAG for everything * **Counting/aggregation:** Store in database (SQL, NoSQL), query directly (e.g., `SELECT COUNT(*) FROM products WHERE company='ACME'` for exact counts) * **Document structure:** Use format-specific APIs/parsers (pandas/openpyxl for spreadsheets, PyPDF2/pdfplumber for PDFs) * **Metadata filtering:** Store metadata in queryable fields (file type, date, author), filter with standard database queries * **Whole-document analysis:** Use long-context models (Claude 3.5 with 200K context) or traditional document processing pipelines * **Principle:** RAG excels at semantic search over unstructured content; use database/API tools built for structured queries, counting, aggregation, format-specific operations #### What it is * Treating all document formats equally in knowledge ingestion * No format-specific processing strategies #### Why it fails * **XLSX files lose row context:** Each cell indexed separately, breaking tabular relationships * **PDFs with complex layouts:** Tables, multi-column text, embedded images get mangled * **Scanned documents:** OCR errors introduce noise * **Presentations:** Slide context and visual hierarchy are lost #### What to do instead: Format-aware processing * **Tabular data:** CSV preferred over XLSX (each row's data preserved in same context) * **Structured data:** Extract to JSON or database format before indexing * **PDFs:** Use layout-aware parsers that preserve table structure * **Presentations:** Extract speaker notes and slide content separately with metadata * **Scanned documents:** Apply OCR with confidence scoring and manual review for critical content #### What it is * Conflating data extraction (getting data as-is) with document processing (analyzing, calculating, transforming) * Cramming extraction AND processing into single Document Extractor node * Assuming Document Extractor handles both pulling data AND performing operations on it #### Why it fails * Document Extractor is prompt-engineered for extraction only (data as it appears on page) * Verbs in entity definitions ("summarize," "calculate") interfere with extraction-focused prompt * Mixing extraction with transformation confuses the model (extract verbatim or perform operations?) * Multi-step logic doesn't fit extraction paradigm (no conditionals, aggregations, or multi-pass analysis) #### What to do instead: Decompose document tasks * **Step 1 - Extract with Document Extractor:** Pull raw data exactly as it appears * **Step 2 - Process with appropriate tools:** Calculations (calculator/code), summarization (separate LLM call), validation (business logic tools), comparison (dedicated tools), aggregation (data processing/SQL) * **Step 3 - Orchestrate with Agentic Workflow:** Coordinate multi-step process (extract → validate → calculate → summarize → return) * Each step uses right tool for the job, orchestrated by workflow layer (more reliable, easier to debug, more maintainable, better edge case handling) #### What it is * Using minimal, vague, or ambiguous field definitions in Document Extractor * Spending minimal effort on field specs and assuming system will figure it out * Examples: `amount` (which amount?), `date` (which date?), `address` (which address?), "the price" (which price?) #### Why it fails * Field names/descriptions become part of the extraction prompt (vague inputs → vague outputs) * Complex documents have multiple similar items that need disambiguation (invoices have 7+ different "amounts") * Semantic ambiguity causes wrong extractions ("date" could mean 5+ different dates on one document) * Works for simple documents but fails at scale (receipt vs. contract/medical record/financial statement) #### What to do instead: Explicit, unambiguous field definitions * **Use specific field names:** `invoice_total_amount` not `amount`; `invoice_date` not `date`; `billing_address` not `address`; `customer_legal_name` not `name` * **Provide detailed descriptions:** "The final total amount due including all line items, taxes, and fees, typically found at the bottom of the invoice" * **Specify location hints:** "The payment due date, usually labeled as 'Due Date' or 'Payment Due By' near the top right of the invoice" * **Disambiguate similar fields:** Explicitly state which of multiple similar items you want (unit\_price vs. extended\_price) * **Include format expectations:** "The date the invoice was issued, in MM/DD/YYYY format, typically labeled 'Invoice Date' near the invoice number" * **Specify data types and constraints:** "List of all items purchased. Each item should include: item\_description (string), quantity (number), unit\_price (decimal), line\_total (decimal)" * **Treat field definitions as prompt engineering:** Quality of extraction directly tied to quality of field specifications ## Related topics [Foundational architecture considerations](/agents/agent_design/foundational_architecture) [Tooling and scalability considerations](/agents/agent_design/tooling_scalability) [watsonx Orchestrate skills](/agents/agent_design/wxo_skills)