Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 19: Model-Based RL
This lecture reviews advanced topics in Reinforcement Learning (RL), transitioning from model-free policy optimization methods (TRPO/PPO) to the critical challenges of Model-Based RL. The core focus is addressing model uncertainty when using learned dynamics for planning. Techniques such as Bayesian statistics, Gaussian Processes (GPs), and Ensembles are introduced to quantify epistemic uncertainty, allowing planners to compute expected rewards by averaging predictions over a posterior distribution of possible models.
Key takeaways
-
PPO/TRPO for Policy Optimization
16:15
Policy optimization methods (like TRPO and PPO) define a surrogate objective function to estimate the policy gradient, enabling continuous updates. PPO uses a clipped ratio ($ ext{clip}(r_{ heta}, 1- ext{eps}, 1+ ext{eps})$) to constrain the new policy's divergence from the old one, stabilizing training without requiring complex second-order optimization.
-
Model-Based RL Limitations
25:00
The basic model-based recipe (collect data $ ightarrow$ fit dynamics $P(s'|s, a)$ $ ightarrow$ plan) fails when dealing with complex or nonlinear dynamics because extrapolation outside the observed state distribution is unreliable. This issue of generalization and distribution shift must be addressed.
-
Quantifying Model Uncertainty
35:00
To improve model-based planning, uncertainty quantification is necessary. The distinction between Aleatoric (inherent noise) and Epistemic (model uncertainty) is crucial. Bayesian approaches treat this by modeling the posterior distribution over parameters ($ heta$), allowing for prediction averaging across all plausible models.
-
Ensemble Methods for Uncertainty
46:40
A practical approach to estimate model uncertainty is using ensembles: training multiple independent neural network copies. The average of their predictions approximates the predictive posterior distribution, effectively exploring multimodal solution landscapes without requiring complex analytical derivations.
