# Build A Reasoning Model From Scratch 8: Distillation

## Executive summary

This video details model distillation, presenting it as a computationally efficient alternative to Reinforcement Learning with Verifiable Rewards (RLVR) for developing reasoning LLMs. The process involves training a smaller 'student model' (e.g., Qwen 3 0.6B) to mimic the outputs of a large, state-of-the-art 'teacher model' (e.g., DeepSeek R1 67B). The methodology utilizes supervised fine-tuning, employing Cross-Entropy Loss over the generated reasoning traces and final answers. The process is significantly less computationally expensive than RLVR, allowing for faster training while maintaining high performance on benchmarks like MATH-500.

## Key takeaways

- Model Distillation vs. RLVR: Distillation is an efficient method to transfer knowledge from a large teacher model to a smaller student model, avoiding the high computational cost of RLVR. The student model learns to mimic the teacher's reasoning process and final answers.
- Distillation Techniques (Soft vs. Hard): Soft distillation trains on full logit/softmax probability distributions, providing more information but requiring the student and teacher to share the same vocabulary and model family. Hard distillation trains only on the final predicted tokens, which is simpler and preferred when vocabularies differ (e.g., DeepSeek R1 to Qwen 3).
- Off-policy vs. On-policy Data Collection: Off-policy distillation collects data (answers and logits) from the teacher model ahead of time, making the process faster. On-policy distillation collects data during the training process using the current student model.
- Training Mechanism: The training uses Cross-Entropy Loss, treating the task as a next-token prediction classification problem. The model is trained to maximize the log probability of the correct token (the teacher's output) at every step of the reasoning trace.

## Technical details

- Data Preparation and Tokenization: The process uses the Math dataset (12,000 training samples). Data is pre-tokenized to save computation time. The input data structure includes the `prompt`, `ground truth answer`, `thinking trace` (message thinking), and `message content` (final answer).
- Loss Function Implementation: The complex GRL loss from RLVR is replaced by standard Cross-Entropy Loss. The loss is computed only over the answer tokens, not the prompt tokens, by shifting the target tokens by one position relative to the input tokens.
- Training Loop Optimization: The training loop is structured similarly to RLVR but replaces the GRL loss with Cross-Entropy Loss. The process involves multiple epochs, gradient clipping, and checkpoint saving. The overall training time is significantly shorter than RLVR (e.g., 2-3 hours vs. 12 hours for comparable steps).
- Evaluation and Performance: The distilled model is evaluated on the held-out Math-500 dataset. Results show that distillation can yield high accuracy, sometimes surpassing RLVR results, especially when the student and teacher models share the same architecture (e.g., Qwen 3).

## Practical implications

- Model distillation provides a cost-effective and computationally milder path to developing high-performing reasoning models compared to full RLVR training.
- The technique allows developers to leverage the knowledge of massive, proprietary teacher models (like DeepSeek R1) to efficiently train smaller, deployable student models (like Qwen 3).
- The ability to use Cross-Entropy Loss simplifies the training pipeline by replacing complex custom loss functions (like GRL loss) with standard, optimized PyTorch functions.

## Topics

Large Language Models (LLMs), Model Distillation, Supervised Fine-Tuning, Reinforcement Learning (RL), Natural Language Processing (NLP), Deep Learning Training, Reasoning playlist, Reasoning Book, Reasoning GitHub repo, LLMs from Scratch book, LLMs from Scratch playlist

Source: https://www.youtube.com/watch?v=msp69FR5NZs