# Stanford CS329A Self-Improving AI Agents | Part 6 | Train Time Scaling/Scaling RL

## Executive summary

This lecture explores advanced techniques for scaling Large Language Model (LLM) reasoning capabilities through 'train-time scaling' and Reinforcement Learning (RL). The discussion covers three key papers: STaR (Self-Taught Reasoner), DeepSeekMath, and DAPO. Key findings demonstrate that smaller models can achieve high accuracy on complex benchmarks like AIME by leveraging structured training data generation (STaR) or advanced RL algorithms (DAPO/GRPO). The core insight is that closing the feedback loop—using model outputs to improve the model itself—is crucial for boosting reasoning, especially in domains with verifiability.

## Key takeaways

- Train-Time Scaling vs. Test-Time Scaling: While test-time scaling (inference-based techniques like majority voting) improves accuracy by sampling outputs, train-time scaling uses the model's own filtered outputs to fine-tune and improve the model weights directly, creating a powerful closed feedback loop.
- STaR Boosts Reasoning via Rationalization: The STaR method bootstraps reasoning by generating solutions on a small set of examples. It filters for correct answers and then generates rationales (hints) for incorrect attempts, allowing the model to learn from failed paths iteratively.
- GRPO Addresses RL Memory Constraints: DeepSeekMath introduced Group Relative Policy Optimization (GRPO), an efficient alternative to PPO that reduces memory overhead by using a group baseline instead of maintaining multiple policy copies, enabling scaling RL to larger models.
- DAPO Stabilizes Complex Reasoning: DAPO addresses training instability in long chain-of-thought reasoning by implementing asymmetric clipping (allowing bigger increases) and dynamic sampling (filtering out zero or one reward groups to maintain a useful gradient signal).

## Technical details

- STaR Algorithm for Reasoning: STaR generates training data by starting with initial examples, filtering only those with correct answers. For incorrect attempts, it uses the known answer as a hint and prompts the model to generate a rationale (explanation), expanding the dataset iteratively.
- DeepSeekMath & GRPO: The DeepSeekMath paper improved mathematical reasoning by starting with a code-pretrained model and curating math data from Common Crawl (OpenWebMath). It utilized Group Relative Policy Optimization (GRPO) to scale RL, achieving 51.7% accuracy on the Math benchmark using only three copies of the model instead of four.
- DAPO Techniques: To stabilize RL for long reasoning chains, DAPO employs asymmetric clipping (allowing greater exploration) and dynamic sampling. Dynamic sampling filters out samples with zero or one reward to ensure the gradient signal is non-zero and useful.
- RL Optimization Challenges: The speaker notes that RL optimization is difficult because the reward model can be 'hacked' if the model becomes too capable, and a lack of signal prevents effective hill climbing. The optimal approach requires controlling not just the loss function, but also response length and entropy.

## Practical implications

- The techniques demonstrate that high performance on complex reasoning tasks (like AIME) can be achieved with smaller, more accessible models by focusing compute on iterative self-improvement rather than solely increasing parameter count.
- For systems requiring verifiable outputs (e.g., code generation, mathematical calculation), implementing a closed feedback loop using model output filtering is critical for boosting reliability and accuracy.
- The need to control multiple variables in RL—including response length penalty and entropy—highlights that robust AI system development requires careful monitoring of training stability alongside performance metrics.

## Topics

Reinforcement Learning (RL), Large Language Models (LLMs), Reasoning Capabilities, Self-Improving Agents, Train-Time Scaling, Model Optimization, Stanford CS329A Self-Improving AI Agents, Agentic AI professional education program

Source: https://www.youtube.com/watch?v=yVnmHSAy3ck