Building retrieval harness for enterprise agents
Summary
This technical walkthrough details the architecture of an enterprise agentic retrieval harness, moving beyond traditional Retrieval Augmented Generation (RAG) systems. The core argument is that effective data access requires giving agents a comprehensive set of tools—including local file system traversal (GP), semantic search, keyword search, and multimodal analysis—rather than relying on a single indexing method. Key focus areas include managing complex document parsing, ensuring multi-tenancy via custom metadata, and optimizing vector storage for scale and cost.
Key takeaways
-
Harness over RAG
2:00
The most capable approach is not to select a single retrieval method (e.g., pure semantic search or pure GP access) but to build an agentic harness that allows the agent to decide how to combine multiple tools for data exploration and verification.
-
The Role of File Primitives
4:20
For document-driven workflows, critical primitives include: (1) Retrieval (hybrid search), (2) Directory listing/hierarchy traversal, (3) Grep for exact text matching, (4) Chunk-by-chunk analysis, and (5) Multimodal processing via localized screenshots.
-
Enterprise Scaling Challenges
6:45
Scaling to multi-tenant or million-file situations requires specialized indexing pipelines that manage data freshness, security, and permissioning. Using custom metadata attached during the parsing process is crucial for filtering access at the vector storage layer.
Technical details
-
Hybrid Search & Retrieval
305s
Combines pure vector-based search (semantic, non-verbatim) with keyword-based search (verbatim similarity). Using a fusion/reranking algorithm on multiple queries improves accuracy and provides the agent with better starting context.
-
Indexing Pipeline Stages
350s
The process is broken into three stages for scalability: (1) Ingesting data from the source, (2) Scaling out the syncing and parsing pipeline (handling complex file types), and (3) Exporting/managing the vector store. Parsing itself requires OCR-based and vision-based LLM capabilities.
-
Vector Storage Architecture
380s
For large, multi-tenant deployments, disk-backed (cheap at rest) vector stores are preferred over RAM-only solutions. The ideal setup uses a hybrid approach: long-term durable data on disk, pulled into faster memory/SSD only when needed.
-
Agentic Tooling and Context Management
150s
To manage the cost of large context windows and prevent token budget overruns, techniques like sub-agent execution are explored. The goal is to provide the agent with a 'steering wheel' of tools rather than relying on monolithic RAG.
Mentioned resources
- LlamaParse Index
- ParsBench
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.
