Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 9: Stochastic Dyn. Program
This lecture details the extension of optimal control theory from deterministic settings to stochastic environments using Markov Decision Processes (MDPs). The core methodology involves defining state transitions and costs that incorporate random disturbances ($w_k$). For finite-horizon problems, the solution relies on adapting the Bellman recursion by taking the expectation over all disturbance realizations. Crucially, for infinite-horizon MDPs—which are foundational to Reinforcement Learning (RL)—the problem is simplified by assuming stationarity and introducing a discount factor ($\gamma$), leading to fixed-point equations for the optimal value function ($V^*$) and the Q-function ($Q^*$).
Key takeaways
-
Stochastic State Dynamics (MDP)
2:00
The state update is modeled as $x_{k+1} = f(x_k, u_k, w_k)$, where $w_k$ is a random disturbance. The system must adhere to the Markovian assumption: the probability distribution of $w_k$ can only depend on the current state ($x_k$) and control ($u_k$), not on the history of previous states or disturbances.
-
Finite-Horizon Optimization
4:00
The notion of optimality is defined by minimizing the expected cost, $\mathbb{E}[ ext{Cost}]$, over all possible disturbance realizations. The solution uses a backward dynamic programming recursion (Bellman equation) to find the optimal closed-loop policy $\pi^*$.
-
Infinite-Horizon MDPs and Discounting
10:20
To solve problems over an infinite number of stages, a discount factor ($\gamma \in [0, 1]$) is introduced to ensure the convergence of the expected cumulative reward. The optimal value function $V^*$ satisfies a fixed-point equation: $V^*(x) = \max_{u} \{ R(x, u) + \gamma \mathbb{E}[V^*(x')]\}$.
-
The Q-Function Formulation
17:30
For computational tractability in learning settings where the transition kernel is unknown, the problem can be reformulated using the Q-function ($Q^*$), which represents the expected cumulative reward starting at state $x$ and taking action $u$, followed by optimal actions: $Q^*(x, u) = R(x, u) + \gamma \sum_{x'} T(x'|x, u) V^*(x')$.
