# Teaching LLMs to Speak Spotify — Yves Raimond & Jacqueline Wood, Spotify

## Executive summary

Spotify has transitioned its recommendation system from traditional curation and ranking algorithms to 'generative personalization' by making Large Language Models (LLMs) native to its catalog. The core technology, the Large Taste Model, allows users to interact with the system using natural language, enabling features like steerable DJs, prompted playlists, and editable taste profiles. The system is powered by a four-stage training recipe called NEO, which embeds open-weight LLMs with Spotify's catalog knowledge using Semantic IDs, ensuring both high performance and the retention of core language abilities.

## Key takeaways

- Shift to Generative Personalization: Spotify moved from 'personalization as guessing' (ranking algorithms) to 'personalization as reasoning,' allowing the system to introspect and generate experiences dynamically shaped around the user's intent. This also represents a shift from black-box algorithms to transparent, steerable systems. (2:37)
- The Large Taste Model (LTM): The LTM is the central system that combines prediction and reasoning, allowing users to shape and generate experiences in real time. As of today, about one in four US Premium subscribers interact with it daily. (6:52, 7:27)
- LLM Judge Grounding for Evaluation: Traditional offline metrics are insufficient for generative systems. To evaluate performance, Spotify grounds LLM judges using textual user profiles (summarizing listening history) and actual behavioral signals. Grounding on ambiguous queries increased alignment with human preferences by 91%. (10:52)

## Technical details

- Semantic ID Implementation: Semantic IDs are discrete tokens created by applying a quantization algorithm to existing content embeddings (e.g., podcast episode embeddings). These IDs are added to an open-weight LLM (like Qwen) vocabulary, allowing the model to understand both natural language and catalog entities. (8:18)
- NEO Training Recipe (Four Stages): The model is trained using NEO: 1) Semantic Foundation (creating tokens); 2) Domain Grounding (freezing the LLM backbone and learning a bidirectional mapping between Semantic IDs and text to prevent catastrophic forgetting); 3) Capability Induction (multitask instruction tuning on specific Spotify tasks); and 4) Optional Post-training (e.g., RL fine-tuning). (9:33)
- Inference Strategy: Testing showed that even without constrained decoding, 98% of generated semantic IDs are valid. While constrained decoding adds latency, beam search was preferred over top P sampling due to significantly higher accuracy. (10:52)

## Practical implications

- The architecture demonstrates a robust pattern for integrating specialized domain knowledge (catalog IDs) into general-purpose LLMs via token embedding and fine-tuning.
- The use of a frozen backbone during domain grounding mitigates catastrophic forgetting, a critical design pattern for maintaining core language ability while adding domain specificity.
- Evaluation pipelines must evolve beyond traditional metrics, requiring the implementation of grounded LLM judges that incorporate user behavior and context to assess recommendation quality and explanation accuracy.

## Topics

LLMs, Generative AI, Recommendation Systems, Natural Language Processing, Semantic IDs, Machine Learning Training, Spotify

Source: https://www.youtube.com/watch?v=2LRIAfng7eA