Stanford CS229 Machine Learning | Spring 2026 | Lecture 18: GMM (EM), PCA
Summary
This lecture provides a deep dive into Reinforcement Learning (RL), focusing on the formal framework of Markov Decision Processes (MDPs) and the Policy Gradient method. The core objective is to solve sequential decision-making problems by maximizing expected cumulative reward. Key concepts include defining states ($S$), actions ($A$), stochastic transition dynamics ($P(s'|s, a)$), and utilizing the Bellman equation for recursive value estimation. The lecture concludes with an explanation of the Policy Gradient algorithm (REINFORCE), detailing how to compute the gradient of the expected return using log-probability tricks, which is crucial for training policies in large models.
Key takeaways
-
Sequential Decision Making & RL Fundamentals
RL addresses sequential decision-making where actions have long-term ramifications. It requires balancing the trade-off between exploitation (using current best knowledge) and exploration (gathering information). Learning relies on maximizing a scalar reward signal rather than explicit labels or supervision.
-
Markov Decision Process (MDP) Framework
4:00
An MDP formally describes an environment using five components: State Set ($S$), Action Set ($A$), Transition Dynamics ($P(s'|s, a)$), Reward Function ($R$), and Discount Factor ($\gamma$). The Markov property ensures that the future state transition depends only on the current state and action, not on history.
-
Value Functions and Bellman Equation
28:50
The value function $V^{\pi}(s)$ estimates the expected total payoff starting at state $s$ under policy $\pi$. The optimal value, $V^*(s)$, is the maximum possible return. These values are solved recursively using the Bellman equation, which relates the current state's value to the expected discounted future rewards.
-
Policy Gradient Method (REINFORCE)
43:20
The Policy Gradient algorithm optimizes a stochastic policy $\pi_{\theta}(a|s)$ by maximizing the expected return $E[R]$. The gradient is computed using the log-probability trick, allowing the calculation of $\nabla_{\theta} E[R]$ through sampling, even when the dependency on $\theta$ only affects the sampling distribution.
Technical details
-
Markov Decision Process (MDP) Components
240s
The MDP is defined by: $S$: Set of states (can be continuous). $A$: Set of actions. $P(s'|s, a)$: Transition probability distribution over the next state $s'$. $R$: Reward function (often $R(s)$ or $R(s, a)$). $\gamma$: Discount factor ($0 o 1$), which discounts future rewards to make the infinite sum of payoffs bounded.
-
Policy Definition and Optimization
1420s
A policy ($\pi$) is a mapping from state $S$ to action $A$. While an optimal policy exists that is deterministic, randomized policies are often used in training for continuity. The goal is finding the optimal policy $\pi^*$ that maximizes expected return.
-
Bellman Equation Recursion
2000s
The Bellman equation provides a recursive relationship to compute value functions, expressing $V^{\pi}(s)$ in terms of immediate rewards and discounted future values. This linear system can be solved for all state values.
-
Policy Gradient Theorem (REINFORCE)
2900s
The gradient $\nabla_{\theta} E[R]$ is computed by rewriting the expectation using log-probabilities: $E[ abla_{\theta} ext{log } p_{\theta}(x)]$. This allows the use of Monte Carlo sampling to estimate the gradient, which is essential for training policies in complex environments like LLMs.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.