# Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex

## Executive summary

The talk outlines the evolution of Retrieval-Augmented Generation (RAG) toward 2026, positing that RAG will decompose into an agent harness and a dedicated context layer. The primary challenge remains unlocking the vast, unstructured context locked in enterprise documents (PDFs, Word, etc.). The solution requires a three-layered platform: 1) Parsing into token-efficient markdown; 2) Semantic storage/document management; and 3) Repeatable, specialized workflows (e.g., KYC, invoice processing). The speaker emphasizes that document OCR is a complex problem due to how PDFs are rendered for display, not machine consumption, necessitating hybrid parsing approaches that combine deep binary understanding with advanced Vision-Language Models (VLMs).

## Key takeaways

- RAG in 2026: Agent Harness + Context Layer: Naive RAG (chunking, embedding, top-k retrieval) is outdated. Modern agents integrate retrieval complexity into the agent layer, allowing them to reason about the optimal search term rather than relying on fixed top-k retrieval methods. (0:00 - 1:33)
- Context Moves Up the Stack: The focus is shifting from managing context window overflow to hooking up specialized Micro-Capabilities Platforms (MCP) and skills. Furthermore, defining programs and tasks is moving from code (Python, TypeScript) toward natural English language definitions. (3:11)
- Document Parsing Requires Hybrid Approaches: Because PDFs are designed for printing (using glyphs and line segments) and not machine reading, document OCR is difficult. The optimal approach combines pipeline-based understanding of file binaries (e.g., Word's custom XML) with VLM-based visual analysis to achieve high accuracy and low cost. (5:15)
- The Three-Layer Document Platform: A comprehensive document context layer requires three components: 1) Parsing into token-efficient markdown/metadata; 2) Semantic storage (document management for agents); and 3) Repeatable, specialized workflows (e.g., claims, invoices). (5:15)

## Technical details

- Document Parsing Challenges: PDFs are rendered for display, meaning text is represented as individual glyphs with coordinates, and tables are not structurally represented. This makes simple binary parsing insufficient. Word and PowerPoint files are also complex, requiring inference of structure beyond native XML tags. (5:15)
- Hybrid Parsing Strategy: The optimal document understanding approach combines pipeline-based methods (deeply understanding file containers) with VLM-based visual approaches. This hybrid model is necessary to balance the trade-offs between cost, accuracy, and latency. (5:15)
- Benchmarking and Scaling: The speaker introduced ParseBench, a public benchmark containing 2,000 human-verified pages, designed to measure how AI agents understand documents across tables, charts, and content faithfulness. It measures performance across different regimes: high accuracy (regulated finance), low cost (indexing millions of documents), and low latency (real-time uploads). (5:15)
- Low Latency Parsing Tooling: For high-throughput, low-latency scenarios (e.g., processing a thousand PDFs in a minute), the speaker recommends LightParse, a free, Rust-based, open-source markdown parser that does not rely on a VLM, serving as a fast initial pass before deeper VLM analysis. (5:15)

## Practical implications

- Build pipelines that incorporate specialized, low-latency parsers (like LightParse) as a fast initial pass, followed by deeper, VLM-enabled tools only when complex visual analysis (tables, charts) is required.
- Design document processing systems with a clear separation between the parsing layer, the semantic storage layer, and the workflow execution layer to ensure modularity and scalability.
- When building agentic applications, focus on enabling the agent to reason about the search query (the 'why') rather than simply providing a fixed top-k retrieval mechanism (the 'what').
- For regulated industries, prioritize high accuracy regimes, even if it increases cost, to minimize the risk of incorrect data extraction.

## Topics

RAG, AI Agents, Document Understanding, Natural Language Processing, Information Retrieval, LlamaIndex, LightParse, ParseBench

Source: https://www.youtube.com/watch?v=RQi7x-navxU