Teaching LLMs to Speak Spotify — Yves Raimond & Jacqueline Wood, Spotify
Summary
Spotify has transitioned its recommendation system from traditional curation and ranking algorithms to 'generative personalization' by making Large Language Models (LLMs) native to its catalog. The core technology, the Large Taste Model, allows users to interact with the system using natural language, enabling features like steerable DJs, prompted playlists, and editable taste profiles. The system is powered by a four-stage training recipe called NEO, which embeds open-weight LLMs with Spotify's catalog knowledge using Semantic IDs, ensuring both high performance and the retention of core language abilities.
Key takeaways
-
Shift to Generative Personalization
3:57
Spotify moved from 'personalization as guessing' (ranking algorithms) to 'personalization as reasoning,' allowing the system to introspect and generate experiences dynamically shaped around the user's intent. This also represents a shift from black-box algorithms to transparent, steerable systems. (2:37)
-
The Large Taste Model (LTM)
10:52
The LTM is the central system that combines prediction and reasoning, allowing users to shape and generate experiences in real time. As of today, about one in four US Premium subscribers interact with it daily. (6:52, 7:27)
-
LLM Judge Grounding for Evaluation
17:32
Traditional offline metrics are insufficient for generative systems. To evaluate performance, Spotify grounds LLM judges using textual user profiles (summarizing listening history) and actual behavioral signals. Grounding on ambiguous queries increased alignment with human preferences by 91%. (10:52)
Technical details
-
Semantic ID Implementation
818s
Semantic IDs are discrete tokens created by applying a quantization algorithm to existing content embeddings (e.g., podcast episode embeddings). These IDs are added to an open-weight LLM (like Qwen) vocabulary, allowing the model to understand both natural language and catalog entities. (8:18)
-
NEO Training Recipe (Four Stages)
933s
The model is trained using NEO: 1) Semantic Foundation (creating tokens); 2) Domain Grounding (freezing the LLM backbone and learning a bidirectional mapping between Semantic IDs and text to prevent catastrophic forgetting); 3) Capability Induction (multitask instruction tuning on specific Spotify tasks); and 4) Optional Post-training (e.g., RL fine-tuning). (9:33)
-
Inference Strategy
1052s
Testing showed that even without constrained decoding, 98% of generated semantic IDs are valid. While constrained decoding adds latency, beam search was preferred over top P sampling due to significantly higher accuracy. (10:52)
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.