# Stanford CS229 Machine Learning | Spring 2026 | Lecture 18: GMM (EM), PCA

## Executive summary

This lecture provides a deep dive into Reinforcement Learning (RL), focusing on the formal framework of Markov Decision Processes (MDPs) and the Policy Gradient method. The core objective is to solve sequential decision-making problems by maximizing expected cumulative reward. Key concepts include defining states ($S$), actions ($A$), stochastic transition dynamics ($P(s'|s, a)$), and utilizing the Bellman equation for recursive value estimation. The lecture concludes with an explanation of the Policy Gradient algorithm (REINFORCE), detailing how to compute the gradient of the expected return using log-probability tricks, which is crucial for training policies in large models.

## Key takeaways

- Sequential Decision Making & RL Fundamentals: RL addresses sequential decision-making where actions have long-term ramifications. It requires balancing the trade-off between exploitation (using current best knowledge) and exploration (gathering information). Learning relies on maximizing a scalar reward signal rather than explicit labels or supervision.
- Markov Decision Process (MDP) Framework: An MDP formally describes an environment using five components: State Set ($S$), Action Set ($A$), Transition Dynamics ($P(s'|s, a)$), Reward Function ($R$), and Discount Factor ($\gamma$). The Markov property ensures that the future state transition depends only on the current state and action, not on history.
- Value Functions and Bellman Equation: The value function $V^{\pi}(s)$ estimates the expected total payoff starting at state $s$ under policy $\pi$. The optimal value, $V^*(s)$, is the maximum possible return. These values are solved recursively using the Bellman equation, which relates the current state's value to the expected discounted future rewards.
- Policy Gradient Method (REINFORCE): The Policy Gradient algorithm optimizes a stochastic policy $\pi_{\theta}(a|s)$ by maximizing the expected return $E[R]$. The gradient is computed using the log-probability trick, allowing the calculation of $\nabla_{\theta} E[R]$ through sampling, even when the dependency on $\theta$ only affects the sampling distribution.

## Technical details

- Markov Decision Process (MDP) Components: The MDP is defined by: $S$: Set of states (can be continuous). $A$: Set of actions. $P(s'|s, a)$: Transition probability distribution over the next state $s'$. $R$: Reward function (often $R(s)$ or $R(s, a)$). $\gamma$: Discount factor ($0 o 1$), which discounts future rewards to make the infinite sum of payoffs bounded.
- Policy Definition and Optimization: A policy ($\pi$) is a mapping from state $S$ to action $A$. While an optimal policy exists that is deterministic, randomized policies are often used in training for continuity. The goal is finding the optimal policy $\pi^*$ that maximizes expected return.
- Bellman Equation Recursion: The Bellman equation provides a recursive relationship to compute value functions, expressing $V^{\pi}(s)$ in terms of immediate rewards and discounted future values. This linear system can be solved for all state values.
- Policy Gradient Theorem (REINFORCE): The gradient $\nabla_{\theta} E[R]$ is computed by rewriting the expectation using log-probabilities: $E[ abla_{\theta} ext{log } p_{\theta}(x)]$. This allows the use of Monte Carlo sampling to estimate the gradient, which is essential for training policies in complex environments like LLMs.

## Practical implications

- The MDP framework provides a rigorous mathematical model for designing and optimizing complex control systems, applicable in robotics and resource management.
- Policy Gradient methods are foundational for training agents (including LLMs) to perform sequential tasks that require long-term planning and decision-making.
- Understanding the Bellman equation allows engineers to analyze system stability and convergence when implementing iterative optimization loops.

## Topics

Reinforcement Learning, Markov Decision Processes (MDP), Optimal Control Theory, Policy Gradient, Stochastic Optimization, Stanford CS229 Machine Learning, Stanford Online AI Programs

Source: https://www.youtube.com/watch?v=xveNBYVTrqw