Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 17: RL Value-Based Methods
This lecture provides a comprehensive review of model-free Reinforcement Learning (RL) value-based methods. The discussion progresses from foundational concepts—distinguishing between prediction and control—to comparing Monte Carlo (MC) and Temporal Difference (TD) learning. Key algorithms covered include SARSA and Q-learning, which are differentiated by their on-policy versus off-policy nature. To scale these methods to high-dimensional state spaces, the necessity of function approximation is introduced, leading into Deep Q Networks (DQN). The lecture concludes by detailing two critical stabilization techniques for DQN: Experience Replay (to decorrelate samples) and using Fixed Q Targets (to stabilize the target value during training).
Key takeaways
-
MC vs. TD Learning Paradigms
17:03
Monte Carlo methods estimate the expected return ($G_t$) by rolling out an episode until a terminal state, requiring full episodes. Temporal Difference (TD) learning improves upon this by using bootstrapping—defining the target as the instantaneous reward plus the discounted future value ($ ext{Reward} + ext{Discounted Future Value}$), allowing for online updates and handling non-terminal environments.
-
On-Policy vs. Off-Policy Learning
23:50
SARSA is an on-policy algorithm, meaning it improves the policy ($ ext{e.g., } ext{epsilon-greedy}$) that is actively used to generate data in the environment. Q-learning is off-policy; it learns about a target optimal policy (the greedy policy) while using data generated by a different behavior policy (also $ ext{epsilon-greedy}$), which is crucial for utilizing historical or simulated data.
-
Scaling with Function Approximation
35:05
To overcome the curse of dimensionality inherent in tabular value function representations, RL methods transition to parametric functions (e.g., neural networks) that approximate $V(s)$ or $Q(s, a)$. This allows generalization across states and controls.
-
DQN Stabilization Techniques
1:03:20
Deep Q Networks (DQN) stabilize learning using two methods: Experience Replay (storing transitions in a buffer to decorrelate samples, satisfying the IID assumption required for regression) and Fixed Q Targets (using a delayed copy of the network parameters ($ ext{Q}_{ ext{target}}$) to prevent the target from being a moving variable during optimization).
