# Hugging Face Journal Club: Direct On-Policy Distillation

## Executive summary

The discussion details a novel technique called Direct On-Policy Distillation for achieving weak-to-strong generalization in large language models. This method proposes an efficient alternative to expensive full Reinforcement Learning (RL) training by leveraging the policy shift observed when training a small model with RL. Specifically, it uses this policy shift as a dense reward signal to train and update a much larger target model (student), significantly reducing computational costs compared to direct RL on the large model.

## Key takeaways

- Weak-to-Strong Generalization via Policy Shift: Instead of directly training a large model with expensive RL, this method measures how an RL run changes a small model's policy (the 'policy shift'). This shift is then used as a dense reward signal to distill knowledge into the larger target model.
- Efficiency Gains: The technique offers substantial cost savings. For example, training a 7B model via RL might take 320 hours, while using distillation from a 1.5B model's policy shift can reduce the estimated time to around 164 hours.
- Methodological Blurring: The process blurs the line between traditional RL and knowledge distillation by combining two types of losses: the policy shift signal (from RL) and a standard KL term, making the overall training setup highly efficient.

## Technical details

- Direct On-Policy Distillation: The core mechanism involves using the policy shift from a small, RL-trained model (the teacher) to guide the training of a larger student model. This is achieved by combining losses derived from this policy shift with standard on-policy distillation terms.
- Model Architecture and Training: The setup requires at least one base small model, one or more teacher models (RL variants), and the large student model. The process is computationally efficient because it can be implemented using PEFTs (Parameter-Efficient Fine-Tuning) to scale up multiple teachers without excessive compute.
- Generalization Scope: The method demonstrates that the distilled knowledge allows the larger model to generalize effectively, even when trained on shorter sequence rollouts (e.g., 2K tokens), suggesting robustness across different input lengths.

## Practical implications

- Significantly reduces the compute resources required for advanced model training (RL).
- Enables weak-to-strong generalization by transferring RL benefits from small, cheaper models to large, high-capacity models.
- Provides a cost-effective pipeline for improving large foundation models without needing full, multi-step RL optimization.

## Topics

Reinforcement Learning (RL), Knowledge Distillation, Large Language Models (LLMs), Weak-to-Strong Generalization, Model Optimization, Weak-to-Strong Generalization via Direct On-Policy Distillation

Source: https://www.youtube.com/watch?v=S5wE1sjyiQY