# Hugging Face Journal Club: Scaling Laws for Pre-training & RL

## Executive summary

The discussion analyzes a paper proposing a joint scaling law for pre-training and Reinforcement Learning (RL), which models how compute allocation across these stages impacts downstream task performance. Key findings suggest that increasing compute allocated to pre-training significantly boosts the model's ability on specific tasks (Pass 1) but has diminishing returns on generalized capability (Pass K). The analysis highlights the importance of optimizing the trade-off between SFT and RL compute budgets.

## Key takeaways

- Pre-training vs. RL Compute Allocation: The primary takeaway is that increasing compute allocated to pre-training leads to higher performance on downstream tasks (Pass 1). Conversely, while RL improves Pass 1, the model's generalized capability (Pass K) remains relatively stable regardless of the pre-training scale.
- Scaling Laws and Model Size: When fixing the total compute budget, training smaller models for longer is generally more effective than attempting to train larger models, contradicting simple Chinchilla scaling assumptions in certain contexts.

## Technical details

- Joint Scaling Law (Pre-training & RL): The paper proposes a joint scaling law governing the relationship between pre-training compute and subsequent RL performance. This moves beyond traditional single-stage scaling laws like Chinchilla's.
- Training Pipeline Stages: The standard pipeline involves three stages: Pre-training $\rightarrow$ Supervised Fine-Guning (SFT) $\rightarrow$ Reinforcement Learning (RL). The compute allocation across these stages is critical for performance.
- Process-Level Supervision in RL: The RL setup used chess as a benchmark, providing step-by-step rewards for every action taken. This contrasts with conventional RL that often only scores on the final trajectory, offering 'process level supervision.'

## Practical implications

- Model developers should treat compute allocation as a multi-stage optimization problem, balancing pre-training investment against RL refinement.
- The analysis suggests that for fixed resources, optimizing the base model size and training duration may be more impactful than simply scaling up to larger parameter counts.
- For domain-specific tasks (e.g., biology), mid-training compute might offer a valuable alternative to full RL if initial pre-trained models are available.

## Topics

Scaling Laws, Large Language Models (LLMs), Reinforcement Learning (RL), Model Optimization, Compute Budgeting, Understanding Reasoning from Pretraining to Post-Training

Source: https://www.youtube.com/watch?v=XB9PlaofL-Q