Distill the LLM, Don't Serve It: Search & Personalization at DoorDash — Raghav Saboo, DoorDash
This talk outlines how DoorDash is leveraging Large Language Models (LLMs) to solve the core problem of marketplace discovery: semantic understanding, rather than just optimizing for engagement. The central architectural pattern is to perform expensive LLM reasoning offline and then distill the resulting knowledge (labels, IDs, memory) into small, fast models for real-time serving. Four key primitives—LLM supervision, semantic IDs, consumer memory, and steerable content generation—are detailed, showing how they improve retrieval (2.3% NDCG lift) and ranking (4–5% MRR lift) while enabling complex, personalized experiences across diverse verticals (grocery, retail, pets).
Key takeaways
-
Semantic Understanding is the Bottleneck
Effective marketplace discovery requires understanding what items mean and what shoppers truly intend, moving beyond simple engagement optimization.
-
Distill LLM Reasoning Offline
10:21
The core strategy is to perform complex LLM reasoning (e.g., generating labels or taxonomies) once, offline, and then distill this knowledge into lightweight, fast models for online serving.
-
Shared Representations Power Multiple Use Cases
Primitives like Semantic IDs and Memory Blocks create shared, reusable representations (text, vectors, graphs) that power retrieval, ranking, and content generation across the entire system.