Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex
The talk outlines the evolution of Retrieval-Augmented Generation (RAG) toward 2026, positing that RAG will decompose into an agent harness and a dedicated context layer. The primary challenge remains unlocking the vast, unstructured context locked in enterprise documents (PDFs, Word, etc.). The solution requires a three-layered platform: 1) Parsing into token-efficient markdown; 2) Semantic storage/document management; and 3) Repeatable, specialized workflows (e.g., KYC, invoice processing). The speaker emphasizes that document OCR is a complex problem due to how PDFs are rendered for display, not machine consumption, necessitating hybrid parsing approaches that combine deep binary understanding with advanced Vision-Language Models (VLMs).
Key takeaways
-
RAG in 2026: Agent Harness + Context Layer
Naive RAG (chunking, embedding, top-k retrieval) is outdated. Modern agents integrate retrieval complexity into the agent layer, allowing them to reason about the optimal search term rather than relying on fixed top-k retrieval methods. (0:00 - 1:33)
-
Context Moves Up the Stack
5:11
The focus is shifting from managing context window overflow to hooking up specialized Micro-Capabilities Platforms (MCP) and skills. Furthermore, defining programs and tasks is moving from code (Python, TypeScript) toward natural English language definitions. (3:11)
-
Document Parsing Requires Hybrid Approaches
8:35
Because PDFs are designed for printing (using glyphs and line segments) and not machine reading, document OCR is difficult. The optimal approach combines pipeline-based understanding of file binaries (e.g., Word's custom XML) with VLM-based visual analysis to achieve high accuracy and low cost. (5:15)
-
The Three-Layer Document Platform
8:35
A comprehensive document context layer requires three components: 1) Parsing into token-efficient markdown/metadata; 2) Semantic storage (document management for agents); and 3) Repeatable, specialized workflows (e.g., claims, invoices). (5:15)