Hugging Face Journal Club: Scaling Laws for Pre-training & RL
The discussion analyzes a paper proposing a joint scaling law for pre-training and Reinforcement Learning (RL), which models how compute allocation across these stages impacts downstream task performance. Key findings suggest that increasing compute allocated to pre-training significantly boosts the model's ability on specific tasks (Pass 1) but has diminishing returns on generalized capability (Pass K). The analysis highlights the importance of optimizing the trade-off between SFT and RL compute budgets.
Key takeaways
-
Pre-training vs. RL Compute Allocation
15:20
The primary takeaway is that increasing compute allocated to pre-training leads to higher performance on downstream tasks (Pass 1). Conversely, while RL improves Pass 1, the model's generalized capability (Pass K) remains relatively stable regardless of the pre-training scale.
-
Scaling Laws and Model Size
24:45
When fixing the total compute budget, training smaller models for longer is generally more effective than attempting to train larger models, contradicting simple Chinchilla scaling assumptions in certain contexts.