# Build A Reasoning Model From Scratch 7: Reinforcement Learning 2 (Tracking, KL Terms, And More)

## Executive summary

This advanced tutorial deep dives into optimizing the Group Relative Policy Optimization (GRPO) algorithm for training reasoning models. Key improvements covered include implementing clipped policy ratios to stabilize training updates, incorporating a KL divergence loss term to constrain model deviation from a reference model, and utilizing specialized format rewards (e.g., think tags) to guide model output structure. The session also provides detailed analysis of advanced training metrics—such as advantage statistics and entropy—and discusses best practices for running long-term, resource-intensive training jobs.

## Key takeaways

- Stabilizing Training with Clipped Policy Ratios: To prevent large, unstable updates (overshooting) during training, the GRPO algorithm can be enhanced by implementing clipped policy ratios. This constrains the size of the policy updates, leading to a more stable training process.
- Advanced Metrics for Diagnosis: Beyond standard loss and reward average, tracking metrics like advantage statistics (mean near zero, high standard deviation) and entropy (measuring model uncertainty) provides deeper insight into the training signal and model behavior.
- Implementing Format Rewards: A format reward can be added to the total reward function (Total Reward = Correctness Reward + Format Reward) to encourage the model to adhere to specific output structures, such as using 'think tags' for reasoning steps.
- KL Divergence for Model Constraint: The KL loss term explicitly limits how much the current model's policy can deviate from a reference model (e.g., the initial model). While powerful, the speaker notes it can be overly complex and is not always necessary, especially in math-based reasoning tasks.

## Technical details

- GRPO Algorithm and Loss Functions: The training uses Group Relative Policy Optimization (GRPO). The total loss is typically a combination of the policy gradient loss, optionally clipped policy ratios, and a KL divergence term. The loss function is minimized to guide the model's policy.
- Advanced Metrics (Advantage and Entropy): Advantage statistics are calculated by standardizing the rewards (Z-score normalization) to center the mean around zero. High standard deviation in advantages indicates a strong, spread-out training signal. Entropy measures the model's uncertainty; a moderate entropy value is desired, avoiding both zero (over-confidence) and maximum (random guessing).
- Resource Management and Scripting: For long-running training jobs (e.g., 500 steps), the speaker recommends using dedicated Python scripts and cloud computing resources (like a separate machine or cloud GPU) rather than running them on a primary machine to prevent system slowdown and overheating.

## Practical implications

- Training complex AI models requires robust resource management; running long, compute-intensive jobs on dedicated hardware (cloud/separate machine) is crucial to maintain system stability and performance.
- Model optimization involves systematically adding and tuning regularization terms (like KL loss or format rewards) to prevent overfitting or reward hacking, requiring careful hyperparameter tuning.
- The analysis of training metrics (advantage, entropy) is critical for diagnosing model instability and determining if the training run is progressing effectively, allowing for targeted algorithmic improvements.

## Topics

Reinforcement Learning (RL), Large Language Models (LLMs), Policy Optimization (GRPO), Training Stability, Information Theory (Entropy), Computational Linguistics, Reasoning playlist, Reasoning Book, Reasoning GitHub repo

Source: https://www.youtube.com/watch?v=VxfUrUrrczs