Stanford CS229 Machine Learning | Spring 2026 | Lecture 20: GMM (EM), PCA
This lecture provides an advanced deep dive into training Large Language Models (LLMs) using Reinforcement Learning (RL). It reviews Policy Gradient methods, detailing the mathematical derivations and limitations. The core focus shifts to Proximal Policy Optimization (PPO), a critical algorithm for stabilizing RL updates by utilizing importance sampling ratios and clipping mechanisms. Finally, the lecture applies these concepts to LLM generation, explaining how Chain-of-Thought (CoT) prompting can be formalized as an MDP problem solved via PPO/SISO.
Key takeaways
-
Policy Gradient Theory
20:40
The policy gradient estimator is necessary because the dependency on parameters ($ heta$) is complex. The fundamental property that $ abla_{ heta} ext{E}_{ ext{P}_{ heta}}[ abla_{ heta} ext{log } ext{P}_{ heta}(a|s)]$ equals zero shows that without a reward function, there are no preferences to optimize for.
-
Proximal Policy Optimization (PPO)
26:40
PPO is designed to stabilize RL training by using importance sampling and clipping the objective function. This prevents the new policy ($ heta$) from deviating too far from the old policy ($ heta_{old}$), which helps maintain stable learning.
-
LLM Generation as an MDP
1:01:40
The LLM generation process is modeled as a Markov Decision Process (MDP). The state ($s_t$) includes the history, and the action ($a_t$) is the next generated token. The reward function is typically applied only at the end of the trajectory based on whether the final answer matches the ground truth.
-
Chain-of-Thought (CoT) Training
1:05:00
To train models for complex reasoning, RL can be used to reward the entire trajectory based on the final answer's correctness. This approach bypasses the need for explicit labeling of the internal 'thinking tokens,' focusing only on verifiable outcomes.
