Build A Reasoning Model From Scratch 8: Distillation
Summary
This video details model distillation, presenting it as a computationally efficient alternative to Reinforcement Learning with Verifiable Rewards (RLVR) for developing reasoning LLMs. The process involves training a smaller 'student model' (e.g., Qwen 3 0.6B) to mimic the outputs of a large, state-of-the-art 'teacher model' (e.g., DeepSeek R1 67B). The methodology utilizes supervised fine-tuning, employing Cross-Entropy Loss over the generated reasoning traces and final answers. The process is significantly less computationally expensive than RLVR, allowing for faster training while maintaining high performance on benchmarks like MATH-500.
Key takeaways
-
Model Distillation vs. RLVR
1:50
Distillation is an efficient method to transfer knowledge from a large teacher model to a smaller student model, avoiding the high computational cost of RLVR. The student model learns to mimic the teacher's reasoning process and final answers.
-
Distillation Techniques (Soft vs. Hard)
4:11
Soft distillation trains on full logit/softmax probability distributions, providing more information but requiring the student and teacher to share the same vocabulary and model family. Hard distillation trains only on the final predicted tokens, which is simpler and preferred when vocabularies differ (e.g., DeepSeek R1 to Qwen 3).
-
Off-policy vs. On-policy Data Collection
7:58
Off-policy distillation collects data (answers and logits) from the teacher model ahead of time, making the process faster. On-policy distillation collects data during the training process using the current student model.
-
Training Mechanism
11:03
The training uses Cross-Entropy Loss, treating the task as a next-token prediction classification problem. The model is trained to maximize the log probability of the correct token (the teacher's output) at every step of the reasoning trace.
Technical details
-
Data Preparation and Tokenization
355s
The process uses the Math dataset (12,000 training samples). Data is pre-tokenized to save computation time. The input data structure includes the `prompt`, `ground truth answer`, `thinking trace` (message thinking), and `message content` (final answer).
-
Loss Function Implementation
663s
The complex GRL loss from RLVR is replaced by standard Cross-Entropy Loss. The loss is computed only over the answer tokens, not the prompt tokens, by shifting the target tokens by one position relative to the input tokens.
-
Training Loop Optimization
750s
The training loop is structured similarly to RLVR but replaces the GRL loss with Cross-Entropy Loss. The process involves multiple epochs, gradient clipping, and checkpoint saving. The overall training time is significantly shorter than RLVR (e.g., 2-3 hours vs. 12 hours for comparable steps).
-
Evaluation and Performance
1150s
The distilled model is evaluated on the held-out Math-500 dataset. Results show that distillation can yield high accuracy, sometimes surpassing RLVR results, especially when the student and teacher models share the same architecture (e.g., Qwen 3).
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.