The Video Signal

Topic

Information Retrieval

All digests tagged Information Retrieval

Distill the LLM, Don't Serve It: Search & Personalization at DoorDash — Raghav Saboo, DoorDash thumbnail

· 22:11

Distill the LLM, Don't Serve It: Search & Personalization at DoorDash — Raghav Saboo, DoorDash

This talk outlines how DoorDash is leveraging Large Language Models (LLMs) to solve the core problem of marketplace discovery: semantic understanding, rather than just optimizing for engagement. The central architectural pattern is to perform expensive LLM reasoning offline and then distill the resulting knowledge (labels, IDs, memory) into small, fast models for real-time serving. Four key primitives—LLM supervision, semantic IDs, consumer memory, and steerable content generation—are detailed, showing how they improve retrieval (2.3% NDCG lift) and ranking (4–5% MRR lift) while enabling complex, personalized experiences across diverse verticals (grocery, retail, pets).

Key takeaways

  1. Semantic Understanding is the Bottleneck

    Effective marketplace discovery requires understanding what items mean and what shoppers truly intend, moving beyond simple engagement optimization.

  2. Distill LLM Reasoning Offline 10:21

    The core strategy is to perform complex LLM reasoning (e.g., generating labels or taxonomies) once, offline, and then distill this knowledge into lightweight, fast models for online serving.

  3. Shared Representations Power Multiple Use Cases

    Primitives like Semantic IDs and Memory Blocks create shared, reusable representations (text, vectors, graphs) that power retrieval, ranking, and content generation across the entire system.

Watch on YouTube Full article

Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex thumbnail

· 21:04

Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex

The talk outlines the evolution of Retrieval-Augmented Generation (RAG) toward 2026, positing that RAG will decompose into an agent harness and a dedicated context layer. The primary challenge remains unlocking the vast, unstructured context locked in enterprise documents (PDFs, Word, etc.). The solution requires a three-layered platform: 1) Parsing into token-efficient markdown; 2) Semantic storage/document management; and 3) Repeatable, specialized workflows (e.g., KYC, invoice processing). The speaker emphasizes that document OCR is a complex problem due to how PDFs are rendered for display, not machine consumption, necessitating hybrid parsing approaches that combine deep binary understanding with advanced Vision-Language Models (VLMs).

Key takeaways

  1. RAG in 2026: Agent Harness + Context Layer

    Naive RAG (chunking, embedding, top-k retrieval) is outdated. Modern agents integrate retrieval complexity into the agent layer, allowing them to reason about the optimal search term rather than relying on fixed top-k retrieval methods. (0:00 - 1:33)

  2. Context Moves Up the Stack 5:11

    The focus is shifting from managing context window overflow to hooking up specialized Micro-Capabilities Platforms (MCP) and skills. Furthermore, defining programs and tasks is moving from code (Python, TypeScript) toward natural English language definitions. (3:11)

  3. Document Parsing Requires Hybrid Approaches 8:35

    Because PDFs are designed for printing (using glyphs and line segments) and not machine reading, document OCR is difficult. The optimal approach combines pipeline-based understanding of file binaries (e.g., Word's custom XML) with VLM-based visual analysis to achieve high accuracy and low cost. (5:15)

  4. The Three-Layer Document Platform 8:35

    A comprehensive document context layer requires three components: 1) Parsing into token-efficient markdown/metadata; 2) Semantic storage (document management for agents); and 3) Repeatable, specialized workflows (e.g., claims, invoices). (5:15)

Watch on YouTube Full article

Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs thumbnail

· 18:01

Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs

This talk challenges the fixed-chunking paradigm in Retrieval-Augmented Generation (RAG) systems. The core claim is that there is no single optimal chunk size; the correct size is query-dependent. The speaker proposes 'multiscale indexing,' which involves duplicating the corpus and chunking it at multiple window sizes. At retrieval time, all resulting indices are queried, and the results are merged using Reciprocal Rank Fusion (RRF), achieving significant recall improvements (20-40%) over fixed-size methods. This approach increases memory usage but maintains low latency overhead.

Key takeaways

  1. Chunking is Query-Dependent, Not Fixed 0:07

    The optimal chunk size varies drastically based on the query type. For example, a highly focused question (e.g., 'Jerry's favorite church') performs best with smaller chunks, while a broad, narrative question (e.g., 'who does Jerry describe as his nemesis') requires larger context windows to capture the answer spread across a scene. (7:07)

  2. Multiscale Indexing is the Solution 0:10

    Instead of committing to one chunk size, the system should index the corpus at multiple window sizes (multiscale indexing). At query time, all these indices are queried, and the resulting rankings are combined using Reciprocal Rank Fusion (RRF). (9:59)

  3. Significant Performance Gains 0:10

    The gap between the performance of the optimal 'oracle' chunk size and any single fixed choice (e.g., 512 tokens) can cost 20% to 40% of recall. The proposed method closes this gap. (9:59)

Watch on YouTube Full article

If we want them to do Knowledge Work, design them as Knowledge Agents — Benjamin Clavié, Mixedbread thumbnail

· 17:55

If we want them to do Knowledge Work, design them as Knowledge Agents — Benjamin Clavié, Mixedbread

The talk distinguishes between 'Coding Agents' and 'Knowledge Agents,' arguing that most real-world tasks fall under the latter. While code provides durable cues (identifiers, file paths), knowledge work is inherently ambiguous, diffuse, and context-dependent (e.g., legal or medical research). The speaker posits that AI agents must be designed to mimic human knowledge work patterns—specifically, through advanced orchestration. This involves breaking down complex, open-ended problems, utilizing multiple specialized tools (primitives like BM25 and semantic search), and employing sub-agents (searchers) to synthesize findings into memos, thereby reducing the 'oracle gap' between perfect knowledge and the agent's output.

Key takeaways

  1. Knowledge Work vs. Coding Work

    Coding is a special, easy case because code has durable cues (identifiers, method definitions). Knowledge work, however, is defined by ambiguous information input and requires reconstructing intent and judgment, making it significantly harder for agents.

  2. The Knowledge Loop 9:15

    Human progress in knowledge work is driven by a single self-optimizing loop: better tools create new roles, new roles generate knowledge, and new knowledge demands better tools. This pattern should guide agent design.

  3. The Necessity of Orchestration

    Effective agent performance requires more than just a single tool. The most significant gains come from architectural improvements, such as having a main agent delegate tasks to specialized 'searcher agents' that return structured memos, which reduces the 'oracle gap' (the difference between perfect information and the system's output) by up to 40%.

  4. Tooling is not Neutral

    Tools are not merely incremental improvements; they are critical for overcoming performance ceilings. The ability to use a tool (e.g., a library catalog vs. physically searching archives) determines if a task is scalable and cheap enough to be practical.

Watch on YouTube Full article

The Search Engine for the Agentic Web — Will Bryk, Exa thumbnail

· 17:49

The Search Engine for the Agentic Web — Will Bryk, Exa

The talk introduces Exa, a search engine designed specifically for AI agents, arguing that traditional search engines (like Google) are fundamentally flawed because they are built for human recommendation rather than precise information retrieval. The core premise is that as AI systems become ubiquitous, the volume of searches issued by machines will surpass human searches by a thousandfold by 2026. Exa addresses this by providing a high-quality, customizable search API that enables agents to perform complex, database-like queries over the world's information, moving beyond simple keyword matching.

Key takeaways

  1. AI Search Volume Prediction

    The speaker predicts that in 2026, the number of searches issued by AI systems will exceed those issued by humans. This volume is expected to increase by a thousandfold in the following years, necessitating a specialized search infrastructure.

  2. The Flaw of Recommendation Engines 5:10

    Mainstream search engines are designed as recommendation engines, meaning they prioritize suggesting related content (e.g., 'shirts with stripes' when 'shirts without stripes' is queried) rather than providing exact, database-level answers. AI agents require the opposite: perfect retrieval.

  3. The Perfect Search Thought Experiment 10:05

    The ideal search involves running a Language Model (LLM) over a complex query and a document pair to determine a match. While this is highly accurate, scaling it to a trillion documents costs millions per query, making cost-optimization the primary engineering challenge.

  4. The Business Catalyst 13:45

    The company's business model was catalyzed when external users requested programmatic API access to their search engine, proving the need for an AI-focused search API, rather than just a consumer product.

Watch on YouTube Full article

The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev thumbnail

· 18:29

The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev

The talk argues that the effectiveness of the lexical scoring function BM25 (Best Match 25) is experiencing a major resurgence due to the emergence of powerful LLM agents. While the scoring function itself has not changed, the agent acts as a vastly more sophisticated user, capable of formulating long, specific queries and executing complex search trajectories. This shift necessitates building retrieval infrastructure that treats the context window as a limited resource (like a floppy disc) and leverages primitives like file system workspaces and `grep` for efficient, end-to-end task completion.

Key takeaways

  1. The User is the Catalyst for BM25's Return 0:03

    LLMs possess general knowledge (entities, companies, dates) that allows them to write queries far more specific and complex than human users. This makes BM25, a 30-year-old lexical function, highly relevant again, moving it beyond a mere baseline.

  2. Agentic Search Requires a Defined Loop

    Agentic search is defined as search within an agent loop, requiring three components: a capable model (able to use tools/formulate queries), a harness (e.g., tool calling or code mode), and an efficient retrieval engine.

  3. Retrieval is Critical for Context Management 0:06

    The context window is limited (compared to a floppy disc, ~1.4 MB). Retrieval is necessary to fetch only the required information from massive document sets, preventing quality degradation and enabling end-to-end task completion.

  4. Evaluation Must Shift to End-to-End Task Completion 0:08

    Traditional evaluation methods based on single-shot queries (like computing NDG) are obsolete. Evaluation must instead focus on whether the model can successfully complete a complex task or answer a riddle-like question using the search tool.

Watch on YouTube Full article

Agents' next frontier: agent-to-agent and network effects — Jean-Denis Greze, Town thumbnail

· 21:17

Agents' next frontier: agent-to-agent and network effects — Jean-Denis Greze, Town

The talk reframes multi-agent systems not as 'agent-to-agent' interactions, but fundamentally as a search problem: ensuring that an LLM's context window contains the optimal information for a tool call. The primary technical barrier to achieving this ideal state—a single agent with access to all world information—is not context length, but privacy and security. Greze outlines five strategies (Shared Trust Boundaries, Custom Tools, Shared Silos, Human Conduit, Black Box) that attempt to approximate the optimal outcome while managing data leakage risks.

Key takeaways

  1. Reframing Agents as Search Problems 2:00

    Most LLM systems are best viewed as search problems. The goal is engineering the system so that the context window contains the precise information needed before a tool call, maximizing the LLM's ability to return the best result.

  2. The Privacy Constraint (Coase Theorem) 5:24

    The ideal state is one agent with access to all world information. However, privacy acts as a transaction cost, preventing this perfect data aggregation, which the Coase theorem highlights.

  3. Shared Silos and Sweeper Agents 13:59

    A promising approach is creating shared silos (e.g., wikis) where a 'sweeper AI' automatically identifies and moves information from private spaces into public, company-wide knowledge bases.

  4. The Black Box Approach

    This advanced method allows an LLM (in a 'black box' agent) to query multiple silos without pinging every human. It only asks the data owners for approval at the final moment of information sharing.

  5. The Future: Auto Mode

    The frontier is 'auto mode,' where LLMs are trusted to automatically determine if a disclosure is low-sensitivity and can be shared without explicit human approval, scaling with model capacity.

Watch on YouTube Full article

Knowledge Systems: The New GTM Stack — Jeffrey Wang, Exa thumbnail

· 18:49

Knowledge Systems: The New GTM Stack — Jeffrey Wang, Exa

Jeffrey Wang argues that Go-To-Market (GTM) strategy must be treated as an AI engineering problem. The core thesis is that GTM is fundamentally a data problem, requiring the creation of a 'live model of your world' that autonomous agents can act upon. He details systems like Exa (a search engine for agents), the ICP dashboard for classifying the Total Addressable Market (TAM), and Request Lens for real-time customer signal detection. Key architectural principles include making the system API-first, recognizing that consistent UIs still complement flexible chatbots, and prioritizing arbitrary customizability over rigid build vs. buy decisions.

Key takeaways

  1. GTM as a Data Problem 4:56

    The goal is to build a live model of the world—combining internal data (customer usage) with external data (web activity, company information)—that agents can programmatically act on. This shifts GTM from a purely sales function to an engineering challenge.

  2. Agent-First Requires API-First 16:59

    For any agent system (whether it's a GUI or a chatbot) to access data, the underlying systems must expose robust programmatic interfaces (APIs). This is critical for enabling agents to function.

  3. System Components: ICP Dashboard & Request Lens 8:38

    The ICP dashboard uses Exa's embeddings over the internet to classify every company in the TAM and estimate anticipated spend. Request Lens provides real-time alerts when significant customer signals occur (e.g., signups, search surges).

  4. The Value of AI Cloning (Jeffbot) 13:42

    An agent can be trained on historical data to mimic a user's professional style and decision-making. Jeffbot was built by analyzing 760 emails and hundreds of past decisions, creating 'evals' to calibrate its judgment against the founder’s own behavior.

Watch on YouTube Full article

How Multi-Vector Retrieval Works at Scale thumbnail

· 24:30

How Multi-Vector Retrieval Works at Scale

This talk introduces Multi-Vector Retrieval, a critical advancement for building sophisticated AI agents and search systems that move beyond the limitations of single vector embeddings. Single vector approaches (which pool token representations) lose low-level detail, making them ineffective for complex, multi-step agentic queries. Multi-vector methods preserve per-token representation, significantly improving retrieval accuracy, especially in out-of-domain or long-context scenarios. The talk details the technical challenges—namely, massive storage and compute overhead—and presents a solution using sparse multi-vector encoding to make billion-document scale retrieval practical.

Key takeaways

  1. Single Vector Limitations for Agents

    Single vector embeddings pool token representations into one summary vector, which captures high-level semantics but loses the low-level detail required for precise queries issued by AI agents. This loss of specificity is a theoretical limit that single vectors cannot overcome, even with increased dimensionality.

  2. Multi-Vector Solution and Scaling 10:15

    Multi-vector embeddings retain one embedding per token instead of pooling them. To manage the resulting storage (10x to 100x increase) and compute overhead, the proposed solution uses sparse multi-vector encoding. This technique approximates MaxSim using random projections, allowing efficient retrieval at scale.

  3. Performance Gains in Agentic Retrieval

    Multi-vector approaches significantly outperform dense models (e.g., a 100M MultiVector model outperforming an 8B dense model) and standard retrieval methods, achieving higher accuracy at substantially lower cost (e.g., 42% accuracy at one thirteenth of the cost).

  4. System Architecture for Scale 20:40

    For production readiness, the system separates compute from storage and isolates read/write paths. This architecture allows handling high write throughput (e.g., 70 MB/s) without negatively impacting query latencies, maintaining sub-50ms P99 latency even at billion scales.

Watch on YouTube Full article

What Is Context Engineering? Why It Matters for AI Agents thumbnail

· 9:57

What Is Context Engineering? Why It Matters for AI Agents

The industry is shifting focus from prompt engineering—which only addresses instruction phrasing—to Context Engineering. Context Engineering is defined as the deliberate practice of structuring and optimizing all information provided to an LLM or AI agent to ensure accurate and reliable outputs. For complex agents that reason across multiple steps, context management is critical because performance can degrade (context rot) when too much irrelevant or poorly structured data is included.

Key takeaways

  1. Context Engineering vs. Prompt Engineering

    Prompt engineering focuses solely on phrasing instructions for an LLM. Context engineering, however, addresses the entire information environment provided to the model, recognizing that prompt engineering is merely one component of context.

  2. The Danger of Over-Context 3:33

    More context does not guarantee better performance. Providing too much irrelevant or poorly structured data can lead to 'context rot,' resulting in worse reasoning and increased hallucinations.

  3. Core Components of Context 4:10

    Context includes the system prompt, user query, retrieved documents (e.g., from a vector database), interaction history, tool outputs, and API results.

Watch on YouTube Full article