Distill the LLM, Don't Serve It: Search & Personalization at DoorDash — Raghav Saboo, DoorDash
Summary
This talk outlines how DoorDash is leveraging Large Language Models (LLMs) to solve the core problem of marketplace discovery: semantic understanding, rather than just optimizing for engagement. The central architectural pattern is to perform expensive LLM reasoning offline and then distill the resulting knowledge (labels, IDs, memory) into small, fast models for real-time serving. Four key primitives—LLM supervision, semantic IDs, consumer memory, and steerable content generation—are detailed, showing how they improve retrieval (2.3% NDCG lift) and ranking (4–5% MRR lift) while enabling complex, personalized experiences across diverse verticals (grocery, retail, pets).
Key takeaways
-
Semantic Understanding is the Bottleneck
Effective marketplace discovery requires understanding what items mean and what shoppers truly intend, moving beyond simple engagement optimization.
-
Distill LLM Reasoning Offline
10:21
The core strategy is to perform complex LLM reasoning (e.g., generating labels or taxonomies) once, offline, and then distill this knowledge into lightweight, fast models for online serving.
-
Shared Representations Power Multiple Use Cases
Primitives like Semantic IDs and Memory Blocks create shared, reusable representations (text, vectors, graphs) that power retrieval, ranking, and content generation across the entire system.
Technical details
-
LLM Supervision for Relevance
327s
To achieve graded relevance (e.g., distinguishing a true gluten-free pasta from a popular regular spaghetti), the system uses a high-quality golden dataset built from human ground truths. This dataset is used to fine-tune a lightweight LLM (e.g., GPD 40 mini) to generate full graded query-item pairs offline. This label is then used to train a two-stage contrastive retrieval method, improving NDCG of relevance by 2.3%.
-
Semantic IDs (Learned Taxonomy)
1100s
Instead of relying solely on coarse human-curated taxonomies, Semantic IDs provide a learned, hierarchical code for catalog items. This allows for cross-category comparisons, aids cold start items, and improves tail coverage by linking sparse items to semantically related neighborhoods. This has improved MRR by 4–5% in ranking and enables query reformulation.
-
Consumer Memory Blocks
Consumer context is represented in three time scales: long-term (durable preferences), real-time (in-session interactions), and stated preferences. Memory is multi-materialized as text, latent vectors, and context graphs, allowing ML models and LLMs to access rich, structured user intent.
-
Steerable Content Generation
Personalized collections are generated offline using LLMs, taking inputs from consumer memory and semantic IDs. This allows the system to 'steer' the content (e.g., generating 'plant-based pantry rows' for a specific affinity) while still using the existing retrieval/ranking stack for item hydration.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.