# Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 19: Model-Based RL

## Executive summary

This lecture reviews advanced topics in Reinforcement Learning (RL), transitioning from model-free policy optimization methods (TRPO/PPO) to the critical challenges of Model-Based RL. The core focus is addressing model uncertainty when using learned dynamics for planning. Techniques such as Bayesian statistics, Gaussian Processes (GPs), and Ensembles are introduced to quantify epistemic uncertainty, allowing planners to compute expected rewards by averaging predictions over a posterior distribution of possible models.

## Key takeaways

- PPO/TRPO for Policy Optimization: Policy optimization methods (like TRPO and PPO) define a surrogate objective function to estimate the policy gradient, enabling continuous updates. PPO uses a clipped ratio ($ ext{clip}(r_{ heta}, 1- ext{eps}, 1+ ext{eps})$) to constrain the new policy's divergence from the old one, stabilizing training without requiring complex second-order optimization.
- Model-Based RL Limitations: The basic model-based recipe (collect data $ ightarrow$ fit dynamics $P(s'|s, a)$ $ ightarrow$ plan) fails when dealing with complex or nonlinear dynamics because extrapolation outside the observed state distribution is unreliable. This issue of generalization and distribution shift must be addressed.
- Quantifying Model Uncertainty: To improve model-based planning, uncertainty quantification is necessary. The distinction between Aleatoric (inherent noise) and Epistemic (model uncertainty) is crucial. Bayesian approaches treat this by modeling the posterior distribution over parameters ($ heta$), allowing for prediction averaging across all plausible models.
- Ensemble Methods for Uncertainty: A practical approach to estimate model uncertainty is using ensembles: training multiple independent neural network copies. The average of their predictions approximates the predictive posterior distribution, effectively exploring multimodal solution landscapes without requiring complex analytical derivations.

## Technical details

- Policy Gradient & Surrogate Objectives: The policy gradient is the expected value of $ abla ext{log } rac{ ext{likelihood}}{ ext{old likelihood}} imes ext{Advantage}$. The importance sampling loss provides an equivalent objective that remains valid even when collecting data using a previous version of the policy.
- Trust Region Policy Optimization (TRPO): TRPO introduces a constraint on the KL divergence between the old and new policies ($ ext{KL}( ext{old} || ext{new})$) to ensure that policy updates remain within a 'trust region' of the original policy.
- Model Predictive Control (MPC): A simple improvement for model-based planning is using MPC, which involves running the plan in a receding horizon fashion and re-evaluating/re-planning based on the current observed state.
- Predictive Posterior Distribution: This is the expected next state predicted by averaging over all possible parameter values ($ heta$) according to the learned posterior distribution, $P(s'|s, a)$. This allows planning under uncertainty.
- Gaussian Processes (GPs): GPs model data as multivariate Gaussians and provide an analytical mechanism to derive the exact posterior distribution over functions, making them highly data efficient in low-data regimes.

## Practical implications

- In autonomous systems, decision pipelines are typically hierarchical: Perception $ ightarrow$ High-level Goal (Dynamic Programming) $ ightarrow$ Open-loop Trajectory Generation $ ightarrow$ Tracking/Control (MPC/PID).
- Model-based RL offers superior sample efficiency compared to model-free methods because learning the dynamics allows the agent to simulate experience internally, reducing real-world interaction time.
- For robust deployment in safety-critical systems, incorporating uncertainty quantification is vital. Planning must account for potential model errors (epistemic uncertainty) by optimizing based on expected rewards across a distribution of possible models.

## Topics

Reinforcement Learning, Optimal Control Theory, Model-Based RL, Policy Optimization, Uncertainty Quantification, Bayesian Inference, Principles of Robot Autonomy, AA203 Optimal and Learning-Based Control Course Info

Source: https://www.youtube.com/watch?v=ZXMThMHFD_w