Hugging Face Journal Club: Direct On-Policy Distillation
The discussion details a novel technique called Direct On-Policy Distillation for achieving weak-to-strong generalization in large language models. This method proposes an efficient alternative to expensive full Reinforcement Learning (RL) training by leveraging the policy shift observed when training a small model with RL. Specifically, it uses this policy shift as a dense reward signal to train and update a much larger target model (student), significantly reducing computational costs compared to direct RL on the large model.
Key takeaways
-
Weak-to-Strong Generalization via Policy Shift
Instead of directly training a large model with expensive RL, this method measures how an RL run changes a small model's policy (the 'policy shift'). This shift is then used as a dense reward signal to distill knowledge into the larger target model.
-
Efficiency Gains
The technique offers substantial cost savings. For example, training a 7B model via RL might take 320 hours, while using distillation from a 1.5B model's policy shift can reduce the estimated time to around 164 hours.
-
Methodological Blurring
The process blurs the line between traditional RL and knowledge distillation by combining two types of losses: the policy shift signal (from RL) and a standard KL term, making the overall training setup highly efficient.