# Stanford CS229 Machine Learning | Spring 2026 | Lecture 20: GMM (EM), PCA

## Executive summary

This lecture provides an advanced deep dive into training Large Language Models (LLMs) using Reinforcement Learning (RL). It reviews Policy Gradient methods, detailing the mathematical derivations and limitations. The core focus shifts to Proximal Policy Optimization (PPO), a critical algorithm for stabilizing RL updates by utilizing importance sampling ratios and clipping mechanisms. Finally, the lecture applies these concepts to LLM generation, explaining how Chain-of-Thought (CoT) prompting can be formalized as an MDP problem solved via PPO/SISO.

## Key takeaways

- Policy Gradient Theory: The policy gradient estimator is necessary because the dependency on parameters ($ heta$) is complex. The fundamental property that $ abla_{ heta} ext{E}_{ ext{P}_{ heta}}[ abla_{ heta} ext{log } ext{P}_{ heta}(a|s)]$ equals zero shows that without a reward function, there are no preferences to optimize for.
- Proximal Policy Optimization (PPO): PPO is designed to stabilize RL training by using importance sampling and clipping the objective function. This prevents the new policy ($ heta$) from deviating too far from the old policy ($ heta_{old}$), which helps maintain stable learning.
- LLM Generation as an MDP: The LLM generation process is modeled as a Markov Decision Process (MDP). The state ($s_t$) includes the history, and the action ($a_t$) is the next generated token. The reward function is typically applied only at the end of the trajectory based on whether the final answer matches the ground truth.
- Chain-of-Thought (CoT) Training: To train models for complex reasoning, RL can be used to reward the entire trajectory based on the final answer's correctness. This approach bypasses the need for explicit labeling of the internal 'thinking tokens,' focusing only on verifiable outcomes.

## Technical details

- Policy Gradient Estimator: The gradient of the return is estimated using a term involving the reward ($R$) and the conditional probability of the whole trajectory. The goal is to maximize the expected return while weighting the log-likelihood by the accumulated reward.
- PPO Objective Function: The PPO objective function uses a ratio ($ ext{ratio} = rac{ ext{P}_{ heta}(a|s)}{ ext{P}_{ heta_{old}}(a|s)}$) and clips the advantage term ($A_t$) to ensure stable updates: $ ext{clip}( ext{ratio} imes A_t, 1- ext{epsilon}_{low}, 1+ ext{epsilon}_{high})$.
- LLM State/Action Dynamics (MDP): The state $s_{t+1}$ is a deterministic concatenation of the previous history and the new action: $s_{t+1} = s_t ext{ concat } a_t$. The reward function is applied only at the end, based on the final answer's match to ground truth.
- Baseline Subtraction: To reduce variance in RL estimation, a baseline ($B(s)$) is subtracted from the reward/advantage function. This baseline can be estimated by averaging rewards across multiple sampled trajectories for the same prompt.

## Practical implications

- Understanding PPO is crucial for implementing stable and robust RL fine-tuning pipelines for LLMs.
- The concept of modeling generation as an MDP allows engineers to apply standard control theory techniques (like reward shaping) to complex generative tasks.
- For build engineering, the ability to programmatically define a 'reward function' based on structured output extraction (e.g., using XML tags or parsers) is key for automated evaluation in CI/CD pipelines.

## Topics

Reinforcement Learning, Large Language Models (LLMs), Policy Gradient, Proximal Policy Optimization (PPO), Chain-of-Thought (CoT), Markov Decision Process (MDP), Stanford CS229 Machine Learning, Stanford AI Programs Info

Source: https://www.youtube.com/watch?v=J7CossjMvEg