Stanford CS229 Machine Learning | Spring 2026 | Lecture 20: GMM (EM), PCA
Summary
This lecture provides an advanced deep dive into training Large Language Models (LLMs) using Reinforcement Learning (RL). It reviews Policy Gradient methods, detailing the mathematical derivations and limitations. The core focus shifts to Proximal Policy Optimization (PPO), a critical algorithm for stabilizing RL updates by utilizing importance sampling ratios and clipping mechanisms. Finally, the lecture applies these concepts to LLM generation, explaining how Chain-of-Thought (CoT) prompting can be formalized as an MDP problem solved via PPO/SISO.
Key takeaways
-
Policy Gradient Theory
20:40
The policy gradient estimator is necessary because the dependency on parameters ($ heta$) is complex. The fundamental property that $ abla_{ heta} ext{E}_{ ext{P}_{ heta}}[ abla_{ heta} ext{log } ext{P}_{ heta}(a|s)]$ equals zero shows that without a reward function, there are no preferences to optimize for.
-
Proximal Policy Optimization (PPO)
26:40
PPO is designed to stabilize RL training by using importance sampling and clipping the objective function. This prevents the new policy ($ heta$) from deviating too far from the old policy ($ heta_{old}$), which helps maintain stable learning.
-
LLM Generation as an MDP
1:01:40
The LLM generation process is modeled as a Markov Decision Process (MDP). The state ($s_t$) includes the history, and the action ($a_t$) is the next generated token. The reward function is typically applied only at the end of the trajectory based on whether the final answer matches the ground truth.
-
Chain-of-Thought (CoT) Training
1:05:00
To train models for complex reasoning, RL can be used to reward the entire trajectory based on the final answer's correctness. This approach bypasses the need for explicit labeling of the internal 'thinking tokens,' focusing only on verifiable outcomes.
Technical details
-
Policy Gradient Estimator
1240s
The gradient of the return is estimated using a term involving the reward ($R$) and the conditional probability of the whole trajectory. The goal is to maximize the expected return while weighting the log-likelihood by the accumulated reward.
-
PPO Objective Function
1600s
The PPO objective function uses a ratio ($ ext{ratio} = rac{ ext{P}_{ heta}(a|s)}{ ext{P}_{ heta_{old}}(a|s)}$) and clips the advantage term ($A_t$) to ensure stable updates: $ ext{clip}( ext{ratio} imes A_t, 1- ext{epsilon}_{low}, 1+ ext{epsilon}_{high})$.
-
LLM State/Action Dynamics (MDP)
3700s
The state $s_{t+1}$ is a deterministic concatenation of the previous history and the new action: $s_{t+1} = s_t ext{ concat } a_t$. The reward function is applied only at the end, based on the final answer's match to ground truth.
-
Baseline Subtraction
1800s
To reduce variance in RL estimation, a baseline ($B(s)$) is subtracted from the reward/advantage function. This baseline can be estimated by averaging rewards across multiple sampled trajectories for the same prompt.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.