# Distill the LLM, Don't Serve It: Search & Personalization at DoorDash — Raghav Saboo, DoorDash

## Executive summary

This talk outlines how DoorDash is leveraging Large Language Models (LLMs) to solve the core problem of marketplace discovery: semantic understanding, rather than just optimizing for engagement. The central architectural pattern is to perform expensive LLM reasoning offline and then distill the resulting knowledge (labels, IDs, memory) into small, fast models for real-time serving. Four key primitives—LLM supervision, semantic IDs, consumer memory, and steerable content generation—are detailed, showing how they improve retrieval (2.3% NDCG lift) and ranking (4–5% MRR lift) while enabling complex, personalized experiences across diverse verticals (grocery, retail, pets).

## Key takeaways

- Semantic Understanding is the Bottleneck: Effective marketplace discovery requires understanding what items mean and what shoppers truly intend, moving beyond simple engagement optimization.
- Distill LLM Reasoning Offline: The core strategy is to perform complex LLM reasoning (e.g., generating labels or taxonomies) once, offline, and then distill this knowledge into lightweight, fast models for online serving.
- Shared Representations Power Multiple Use Cases: Primitives like Semantic IDs and Memory Blocks create shared, reusable representations (text, vectors, graphs) that power retrieval, ranking, and content generation across the entire system.

## Technical details

- LLM Supervision for Relevance: To achieve graded relevance (e.g., distinguishing a true gluten-free pasta from a popular regular spaghetti), the system uses a high-quality golden dataset built from human ground truths. This dataset is used to fine-tune a lightweight LLM (e.g., GPD 40 mini) to generate full graded query-item pairs offline. This label is then used to train a two-stage contrastive retrieval method, improving NDCG of relevance by 2.3%.
- Semantic IDs (Learned Taxonomy): Instead of relying solely on coarse human-curated taxonomies, Semantic IDs provide a learned, hierarchical code for catalog items. This allows for cross-category comparisons, aids cold start items, and improves tail coverage by linking sparse items to semantically related neighborhoods. This has improved MRR by 4–5% in ranking and enables query reformulation.
- Consumer Memory Blocks: Consumer context is represented in three time scales: long-term (durable preferences), real-time (in-session interactions), and stated preferences. Memory is multi-materialized as text, latent vectors, and context graphs, allowing ML models and LLMs to access rich, structured user intent.
- Steerable Content Generation: Personalized collections are generated offline using LLMs, taking inputs from consumer memory and semantic IDs. This allows the system to 'steer' the content (e.g., generating 'plant-based pantry rows' for a specific affinity) while still using the existing retrieval/ranking stack for item hydration.

## Practical implications

- The shift from optimizing for engagement signals to understanding semantic intent is crucial for scaling e-commerce discovery.
- The pattern of 'reasoning offline, serving cheaply' is a scalable architectural pattern for integrating LLMs into production systems.
- By creating shared, structured representations (like Semantic IDs and Memory Blocks), multiple downstream models (retrieval, ranking, generation) can benefit from the same source of truth.

## Topics

LLMs, Search Engineering, Personalization, Marketplace Discovery, Information Retrieval, https://buildshipai.substack.com/ (Substack)

Source: https://www.youtube.com/watch?v=ACPEpji5NV4