Training Agents 3: Reinforcement Learning
Summary
This session introduces Group Relative Policy Optimization (GRPO), a reinforcement learning (RL) method that advances agent training beyond mere imitation (SFT/Distillation). GRPO trains models by sampling multiple completions per prompt and using the group's relative scores—calculated via a reward function—as the primary training signal. This approach is highly effective for complex tasks, allowing agents to learn from their own varied trajectories in an iterative loop.
Key takeaways
-
GRPO Mechanics
20:30
GRPO calculates advantages relative to the group average (Reward - Group Average / Group Standard Deviation). This method eliminates the need for a separate value model, reward model, or critic, simplifying the RL loop. The process involves generating multiple rollouts, scoring them with a verifiable Python function (the reward function), and updating the policy based on these relative advantages.
-
Training Pipeline Progression
5:05
The training pipeline typically progresses from Supervised Fine-Tuning (SFT) for dense, off-policy signals, to Distillation for richer online rollouts, and finally to RL/GRPO for sparse, on-policy learning. SFT is often used first to bootstrap the model's understanding of the task structure.
-
Reward Function Design
26:40
Defining a verifiable reward function is critical; it acts as a 'contract' defining success. Rewards can be composed of multiple components, such as a format check (e.g., ensuring JSON structure) and an accuracy check (e.g., passing unit tests). For agentic tasks, using test suites or compiling code provides robust signals.
-
Interpreting Training Curves
35:00
Monitoring training curves via tools like Track.io is essential for debugging. Key metrics include the reward (should rise), entropy (should remain stable/flat, not dive or spike), and completion length. Failure modes—such as 'reward hacking' (high reward but low test accuracy) or 'collapse' (low entropy)—require deep data inspection.
Technical details
-
Group Relative Policy Optimization (GRPO)
1230s
GRPO is an RL algorithm that updates model weights using relative advantages derived from a group of generated rollouts. It requires defining the reward function as a verifiable Python function, which can combine multiple signals like format and accuracy.
-
RL Guard Rails
1400s
To prevent the model from deviating too far from its base capabilities, two guard rails are used: KL divergence limit (prevents deviation from initial policy) and clipping/batch limit (bounds variation within a specific batch). These parameters ($\epsilon$ and $\beta$) must be carefully tuned.
-
TRL Implementation
1230s
The training process is abstracted by the TRL (Transformer Reinforcement Learning) library. The core loop involves generating a group of responses, calculating rewards/advantages, and updating the policy using a loss function based on advantages times sequence log probability.
-
Failure Modes
2400s
Reward hacking occurs when the model finds an unintended way to maximize the reward signal without solving the actual task (e.g., optimizing for code block count instead of correctness). This requires reviewing the data and reward function, not just relying on KL penalties.
Mentioned resources
- Training Agents Repo
- HF YouTube Channel
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.