Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 19: Model-Based RL
Summary
This lecture reviews advanced topics in Reinforcement Learning (RL), transitioning from model-free policy optimization methods (TRPO/PPO) to the critical challenges of Model-Based RL. The core focus is addressing model uncertainty when using learned dynamics for planning. Techniques such as Bayesian statistics, Gaussian Processes (GPs), and Ensembles are introduced to quantify epistemic uncertainty, allowing planners to compute expected rewards by averaging predictions over a posterior distribution of possible models.
Key takeaways
-
PPO/TRPO for Policy Optimization
16:15
Policy optimization methods (like TRPO and PPO) define a surrogate objective function to estimate the policy gradient, enabling continuous updates. PPO uses a clipped ratio ($ ext{clip}(r_{ heta}, 1- ext{eps}, 1+ ext{eps})$) to constrain the new policy's divergence from the old one, stabilizing training without requiring complex second-order optimization.
-
Model-Based RL Limitations
25:00
The basic model-based recipe (collect data $ ightarrow$ fit dynamics $P(s'|s, a)$ $ ightarrow$ plan) fails when dealing with complex or nonlinear dynamics because extrapolation outside the observed state distribution is unreliable. This issue of generalization and distribution shift must be addressed.
-
Quantifying Model Uncertainty
35:00
To improve model-based planning, uncertainty quantification is necessary. The distinction between Aleatoric (inherent noise) and Epistemic (model uncertainty) is crucial. Bayesian approaches treat this by modeling the posterior distribution over parameters ($ heta$), allowing for prediction averaging across all plausible models.
-
Ensemble Methods for Uncertainty
46:40
A practical approach to estimate model uncertainty is using ensembles: training multiple independent neural network copies. The average of their predictions approximates the predictive posterior distribution, effectively exploring multimodal solution landscapes without requiring complex analytical derivations.
Technical details
-
Policy Gradient & Surrogate Objectives
600s
The policy gradient is the expected value of $ abla ext{log } rac{ ext{likelihood}}{ ext{old likelihood}} imes ext{Advantage}$. The importance sampling loss provides an equivalent objective that remains valid even when collecting data using a previous version of the policy.
-
Trust Region Policy Optimization (TRPO)
1050s
TRPO introduces a constraint on the KL divergence between the old and new policies ($ ext{KL}( ext{old} || ext{new})$) to ensure that policy updates remain within a 'trust region' of the original policy.
-
Model Predictive Control (MPC)
1650s
A simple improvement for model-based planning is using MPC, which involves running the plan in a receding horizon fashion and re-evaluating/re-planning based on the current observed state.
-
Predictive Posterior Distribution
2300s
This is the expected next state predicted by averaging over all possible parameter values ($ heta$) according to the learned posterior distribution, $P(s'|s, a)$. This allows planning under uncertainty.
-
Gaussian Processes (GPs)
2500s
GPs model data as multivariate Gaussians and provide an analytical mechanism to derive the exact posterior distribution over functions, making them highly data efficient in low-data regimes.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.