Hugging Face Journal Club: Scaling Laws for Pre-training & RL

Summary

The discussion analyzes a paper proposing a joint scaling law for pre-training and Reinforcement Learning (RL), which models how compute allocation across these stages impacts downstream task performance. Key findings suggest that increasing compute allocated to pre-training significantly boosts the model's ability on specific tasks (Pass 1) but has diminishing returns on generalized capability (Pass K). The analysis highlights the importance of optimizing the trade-off between SFT and RL compute budgets.

Technical details

• Joint Scaling Law (Pre-training & RL) 120s

  The paper proposes a joint scaling law governing the relationship between pre-training compute and subsequent RL performance. This moves beyond traditional single-stage scaling laws like Chinchilla's.

• Training Pipeline Stages 75s

  The standard pipeline involves three stages: Pre-training $\rightarrow$ Supervised Fine-Guning (SFT) $\rightarrow$ Reinforcement Learning (RL). The compute allocation across these stages is critical for performance.

• Process-Level Supervision in RL 260s

  The RL setup used chess as a benchmark, providing step-by-step rewards for every action taken. This contrasts with conventional RL that often only scores on the final trajectory, offering 'process level supervision.'

Mentioned resources

• Understanding Reasoning from Pretraining to Post-Training (Research Paper)

Channel & topics

• Hugging Face
• Scaling Laws
• Large Language Models (LLMs)
• Reinforcement Learning (RL)
• Model Optimization
• Compute Budgeting
• Understanding Reasoning from Pretraining to Post-Training