Stanford CS229 Machine Learning | Spring 2026 | Lecture 18: GMM (EM), PCA
This lecture provides a deep dive into Reinforcement Learning (RL), focusing on the formal framework of Markov Decision Processes (MDPs) and the Policy Gradient method. The core objective is to solve sequential decision-making problems by maximizing expected cumulative reward. Key concepts include defining states ($S$), actions ($A$), stochastic transition dynamics ($P(s'|s, a)$), and utilizing the Bellman equation for recursive value estimation. The lecture concludes with an explanation of the Policy Gradient algorithm (REINFORCE), detailing how to compute the gradient of the expected return using log-probability tricks, which is crucial for training policies in large models.
Key takeaways
-
Sequential Decision Making & RL Fundamentals
RL addresses sequential decision-making where actions have long-term ramifications. It requires balancing the trade-off between exploitation (using current best knowledge) and exploration (gathering information). Learning relies on maximizing a scalar reward signal rather than explicit labels or supervision.
-
Markov Decision Process (MDP) Framework
4:00
An MDP formally describes an environment using five components: State Set ($S$), Action Set ($A$), Transition Dynamics ($P(s'|s, a)$), Reward Function ($R$), and Discount Factor ($\gamma$). The Markov property ensures that the future state transition depends only on the current state and action, not on history.
-
Value Functions and Bellman Equation
28:50
The value function $V^{\pi}(s)$ estimates the expected total payoff starting at state $s$ under policy $\pi$. The optimal value, $V^*(s)$, is the maximum possible return. These values are solved recursively using the Bellman equation, which relates the current state's value to the expected discounted future rewards.
-
Policy Gradient Method (REINFORCE)
43:20
The Policy Gradient algorithm optimizes a stochastic policy $\pi_{\theta}(a|s)$ by maximizing the expected return $E[R]$. The gradient is computed using the log-probability trick, allowing the calculation of $\nabla_{\theta} E[R]$ through sampling, even when the dependency on $\theta$ only affects the sampling distribution.