Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 17: RL Value-Based Methods
Summary
This lecture provides a comprehensive review of model-free Reinforcement Learning (RL) value-based methods. The discussion progresses from foundational concepts—distinguishing between prediction and control—to comparing Monte Carlo (MC) and Temporal Difference (TD) learning. Key algorithms covered include SARSA and Q-learning, which are differentiated by their on-policy versus off-policy nature. To scale these methods to high-dimensional state spaces, the necessity of function approximation is introduced, leading into Deep Q Networks (DQN). The lecture concludes by detailing two critical stabilization techniques for DQN: Experience Replay (to decorrelate samples) and using Fixed Q Targets (to stabilize the target value during training).
Key takeaways
-
MC vs. TD Learning Paradigms
17:03
Monte Carlo methods estimate the expected return ($G_t$) by rolling out an episode until a terminal state, requiring full episodes. Temporal Difference (TD) learning improves upon this by using bootstrapping—defining the target as the instantaneous reward plus the discounted future value ($ ext{Reward} + ext{Discounted Future Value}$), allowing for online updates and handling non-terminal environments.
-
On-Policy vs. Off-Policy Learning
23:50
SARSA is an on-policy algorithm, meaning it improves the policy ($ ext{e.g., } ext{epsilon-greedy}$) that is actively used to generate data in the environment. Q-learning is off-policy; it learns about a target optimal policy (the greedy policy) while using data generated by a different behavior policy (also $ ext{epsilon-greedy}$), which is crucial for utilizing historical or simulated data.
-
Scaling with Function Approximation
35:05
To overcome the curse of dimensionality inherent in tabular value function representations, RL methods transition to parametric functions (e.g., neural networks) that approximate $V(s)$ or $Q(s, a)$. This allows generalization across states and controls.
-
DQN Stabilization Techniques
1:03:20
Deep Q Networks (DQN) stabilize learning using two methods: Experience Replay (storing transitions in a buffer to decorrelate samples, satisfying the IID assumption required for regression) and Fixed Q Targets (using a delayed copy of the network parameters ($ ext{Q}_{ ext{target}}$) to prevent the target from being a moving variable during optimization).
Technical details
-
SARSA Update Rule (On-Policy)
1270s
The update rule for SARSA uses the quintuple $(S_t, A_t, R_{t+1}, S_{t+1}, A_{t+1})$ and updates $Q(S_t, A_t)$ towards $ ext{Reward} + ext{Discounted Value of } Q(S_{t+1}, A_{t+1})$. This is an on-policy update because the next action $A_{t+1}$ is chosen according to the current policy.
-
Q-Learning Update Rule (Off-Policy)
2035s
The Q-learning target updates $Q(S_t, A_t)$ towards $ ext{Reward} + ext{Discounted Maximum } Q(S_{t+1}, a)$. The key difference is that the next action $a$ is chosen via $ ext{argmax}_a Q(S_{t+1}, a)$, decoupling the target policy from the behavior policy.
-
Deep Q Network (DQN) Architecture
3400s
The DQN uses Convolutional Neural Networks (CNNs) to process raw image inputs (the state $X$) and outputs a single value for every possible action, representing the estimated $Q(X, u)$ for all controls $u$. The learning objective is minimizing the error between the predicted Q-value and the target derived from TD or MC methods.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.