# Building retrieval harness for enterprise agents

## Executive summary

This technical walkthrough details the architecture of an enterprise agentic retrieval harness, moving beyond traditional Retrieval Augmented Generation (RAG) systems. The core argument is that effective data access requires giving agents a comprehensive set of tools—including local file system traversal (GP), semantic search, keyword search, and multimodal analysis—rather than relying on a single indexing method. Key focus areas include managing complex document parsing, ensuring multi-tenancy via custom metadata, and optimizing vector storage for scale and cost.

## Key takeaways

- Harness over RAG: The most capable approach is not to select a single retrieval method (e.g., pure semantic search or pure GP access) but to build an agentic harness that allows the agent to decide how to combine multiple tools for data exploration and verification.
- The Role of File Primitives: For document-driven workflows, critical primitives include: (1) Retrieval (hybrid search), (2) Directory listing/hierarchy traversal, (3) Grep for exact text matching, (4) Chunk-by-chunk analysis, and (5) Multimodal processing via localized screenshots.
- Enterprise Scaling Challenges: Scaling to multi-tenant or million-file situations requires specialized indexing pipelines that manage data freshness, security, and permissioning. Using custom metadata attached during the parsing process is crucial for filtering access at the vector storage layer.

## Technical details

- Hybrid Search & Retrieval: Combines pure vector-based search (semantic, non-verbatim) with keyword-based search (verbatim similarity). Using a fusion/reranking algorithm on multiple queries improves accuracy and provides the agent with better starting context.
- Indexing Pipeline Stages: The process is broken into three stages for scalability: (1) Ingesting data from the source, (2) Scaling out the syncing and parsing pipeline (handling complex file types), and (3) Exporting/managing the vector store. Parsing itself requires OCR-based and vision-based LLM capabilities.
- Vector Storage Architecture: For large, multi-tenant deployments, disk-backed (cheap at rest) vector stores are preferred over RAM-only solutions. The ideal setup uses a hybrid approach: long-term durable data on disk, pulled into faster memory/SSD only when needed.
- Agentic Tooling and Context Management: To manage the cost of large context windows and prevent token budget overruns, techniques like sub-agent execution are explored. The goal is to provide the agent with a 'steering wheel' of tools rather than relying on monolithic RAG.

## Practical implications

- When designing an agent, prioritize giving it a suite of tools (GP, listing, search) over relying solely on pre-indexed vector retrieval.
- For enterprise deployments, implement custom metadata tagging during the ingestion process to enforce permissioning and filtering at the vector store level.
- To handle complex documents (e.g., legal filings), use specialized parsing that extracts localized screenshots alongside raw text, allowing LLMs' vision capabilities to read non-textual context.
- When dealing with relational data, consider exporting structured entries into a standardized format like JSON or plain text before vectorization.

## Topics

Agentic Workflows, Retrieval Augmented Generation (RAG), Vector Databases, Build Engineering, Data Orchestration, LLM Tooling, LlamaParse Index, ParsBench

Source: https://www.youtube.com/watch?v=bXDsohiwdYQ