# Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 17: RL Value-Based Methods

## Executive summary

This lecture provides a comprehensive review of model-free Reinforcement Learning (RL) value-based methods. The discussion progresses from foundational concepts—distinguishing between prediction and control—to comparing Monte Carlo (MC) and Temporal Difference (TD) learning. Key algorithms covered include SARSA and Q-learning, which are differentiated by their on-policy versus off-policy nature. To scale these methods to high-dimensional state spaces, the necessity of function approximation is introduced, leading into Deep Q Networks (DQN). The lecture concludes by detailing two critical stabilization techniques for DQN: Experience Replay (to decorrelate samples) and using Fixed Q Targets (to stabilize the target value during training).

## Key takeaways

- MC vs. TD Learning Paradigms: Monte Carlo methods estimate the expected return ($G_t$) by rolling out an episode until a terminal state, requiring full episodes. Temporal Difference (TD) learning improves upon this by using bootstrapping—defining the target as the instantaneous reward plus the discounted future value ($ ext{Reward} + ext{Discounted Future Value}$), allowing for online updates and handling non-terminal environments.
- On-Policy vs. Off-Policy Learning: SARSA is an on-policy algorithm, meaning it improves the policy ($ ext{e.g., } ext{epsilon-greedy}$) that is actively used to generate data in the environment. Q-learning is off-policy; it learns about a target optimal policy (the greedy policy) while using data generated by a different behavior policy (also $ ext{epsilon-greedy}$), which is crucial for utilizing historical or simulated data.
- Scaling with Function Approximation: To overcome the curse of dimensionality inherent in tabular value function representations, RL methods transition to parametric functions (e.g., neural networks) that approximate $V(s)$ or $Q(s, a)$. This allows generalization across states and controls.
- DQN Stabilization Techniques: Deep Q Networks (DQN) stabilize learning using two methods: Experience Replay (storing transitions in a buffer to decorrelate samples, satisfying the IID assumption required for regression) and Fixed Q Targets (using a delayed copy of the network parameters ($ ext{Q}_{ ext{target}}$) to prevent the target from being a moving variable during optimization).

## Technical details

- SARSA Update Rule (On-Policy): The update rule for SARSA uses the quintuple $(S_t, A_t, R_{t+1}, S_{t+1}, A_{t+1})$ and updates $Q(S_t, A_t)$ towards $ ext{Reward} + ext{Discounted Value of } Q(S_{t+1}, A_{t+1})$. This is an on-policy update because the next action $A_{t+1}$ is chosen according to the current policy.
- Q-Learning Update Rule (Off-Policy): The Q-learning target updates $Q(S_t, A_t)$ towards $ ext{Reward} + ext{Discounted Maximum } Q(S_{t+1}, a)$. The key difference is that the next action $a$ is chosen via $ ext{argmax}_a Q(S_{t+1}, a)$, decoupling the target policy from the behavior policy.
- Deep Q Network (DQN) Architecture: The DQN uses Convolutional Neural Networks (CNNs) to process raw image inputs (the state $X$) and outputs a single value for every possible action, representing the estimated $Q(X, u)$ for all controls $u$. The learning objective is minimizing the error between the predicted Q-value and the target derived from TD or MC methods.

## Practical implications

- The principles of model-free RL are foundational for developing autonomous systems (e.g., robotics, self-driving cars) that must learn optimal behavior without explicit knowledge of system dynamics.
- Understanding the difference between on-policy (SARSA) and off-policy (Q-learning) methods is critical when designing learning agents where data collection might be expensive or unsafe in the real world.
- The techniques used in DQN (Experience Replay, Target Networks) are essential for stabilizing deep RL training, enabling complex AI systems to learn from massive amounts of correlated environmental data.

## Topics

Reinforcement Learning, Optimal Control, Model-Free Methods, Value Iteration, Deep Learning (DQN), On-Policy/Off-Policy Learning, Principles of Robot Autonomy, AA203 Optimal and Learning-Based Control Course Info

Source: https://www.youtube.com/watch?v=zanXSI7zSws