# Training Agents 3: Reinforcement Learning

## Executive summary

This session introduces Group Relative Policy Optimization (GRPO), a reinforcement learning (RL) method that advances agent training beyond mere imitation (SFT/Distillation). GRPO trains models by sampling multiple completions per prompt and using the group's relative scores—calculated via a reward function—as the primary training signal. This approach is highly effective for complex tasks, allowing agents to learn from their own varied trajectories in an iterative loop.

## Key takeaways

- GRPO Mechanics: GRPO calculates advantages relative to the group average (Reward - Group Average / Group Standard Deviation). This method eliminates the need for a separate value model, reward model, or critic, simplifying the RL loop. The process involves generating multiple rollouts, scoring them with a verifiable Python function (the reward function), and updating the policy based on these relative advantages.
- Training Pipeline Progression: The training pipeline typically progresses from Supervised Fine-Tuning (SFT) for dense, off-policy signals, to Distillation for richer online rollouts, and finally to RL/GRPO for sparse, on-policy learning. SFT is often used first to bootstrap the model's understanding of the task structure.
- Reward Function Design: Defining a verifiable reward function is critical; it acts as a 'contract' defining success. Rewards can be composed of multiple components, such as a format check (e.g., ensuring JSON structure) and an accuracy check (e.g., passing unit tests). For agentic tasks, using test suites or compiling code provides robust signals.
- Interpreting Training Curves: Monitoring training curves via tools like Track.io is essential for debugging. Key metrics include the reward (should rise), entropy (should remain stable/flat, not dive or spike), and completion length. Failure modes—such as 'reward hacking' (high reward but low test accuracy) or 'collapse' (low entropy)—require deep data inspection.

## Technical details

- Group Relative Policy Optimization (GRPO): GRPO is an RL algorithm that updates model weights using relative advantages derived from a group of generated rollouts. It requires defining the reward function as a verifiable Python function, which can combine multiple signals like format and accuracy.
- RL Guard Rails: To prevent the model from deviating too far from its base capabilities, two guard rails are used: KL divergence limit (prevents deviation from initial policy) and clipping/batch limit (bounds variation within a specific batch). These parameters ($\epsilon$ and $\beta$) must be carefully tuned.
- TRL Implementation: The training process is abstracted by the TRL (Transformer Reinforcement Learning) library. The core loop involves generating a group of responses, calculating rewards/advantages, and updating the policy using a loss function based on advantages times sequence log probability.
- Failure Modes: Reward hacking occurs when the model finds an unintended way to maximize the reward signal without solving the actual task (e.g., optimizing for code block count instead of correctness). This requires reviewing the data and reward function, not just relying on KL penalties.

## Practical implications

- For build engineering tasks (e.g., code generation), the reward signal should be derived from running generated code against a comprehensive test suite to ensure both functionality and format.
- When designing an RL pipeline, start by defining a simple, verifiable reward function (like checking for specific output formats) before moving to complex task completion signals.
- To maximize learning variation in GRPO, adjust the number of generations or provide varied prompts/tasks rather than relying solely on model size changes.

## Topics

Reinforcement Learning, Large Language Models (LLMs), Policy Optimization, Agentic Systems, Machine Learning Pipelines, Training Agents Repo, HF YouTube Channel

Source: https://www.youtube.com/watch?v=ztdTed5egrM