# Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 9: Stochastic Dyn. Program

## Executive summary

This lecture details the extension of optimal control theory from deterministic settings to stochastic environments using Markov Decision Processes (MDPs). The core methodology involves defining state transitions and costs that incorporate random disturbances ($w_k$). For finite-horizon problems, the solution relies on adapting the Bellman recursion by taking the expectation over all disturbance realizations. Crucially, for infinite-horizon MDPs—which are foundational to Reinforcement Learning (RL)—the problem is simplified by assuming stationarity and introducing a discount factor ($\gamma$), leading to fixed-point equations for the optimal value function ($V^*$) and the Q-function ($Q^*$).

## Key takeaways

- Stochastic State Dynamics (MDP): The state update is modeled as $x_{k+1} = f(x_k, u_k, w_k)$, where $w_k$ is a random disturbance. The system must adhere to the Markovian assumption: the probability distribution of $w_k$ can only depend on the current state ($x_k$) and control ($u_k$), not on the history of previous states or disturbances.
- Finite-Horizon Optimization: The notion of optimality is defined by minimizing the expected cost, $\mathbb{E}[ ext{Cost}]$, over all possible disturbance realizations. The solution uses a backward dynamic programming recursion (Bellman equation) to find the optimal closed-loop policy $\pi^*$.
- Infinite-Horizon MDPs and Discounting: To solve problems over an infinite number of stages, a discount factor ($\gamma \in [0, 1]$) is introduced to ensure the convergence of the expected cumulative reward. The optimal value function $V^*$ satisfies a fixed-point equation: $V^*(x) = \max_{u} \{ R(x, u) + \gamma \mathbb{E}[V^*(x')]\}$.
- The Q-Function Formulation: For computational tractability in learning settings where the transition kernel is unknown, the problem can be reformulated using the Q-function ($Q^*$), which represents the expected cumulative reward starting at state $x$ and taking action $u$, followed by optimal actions: $Q^*(x, u) = R(x, u) + \gamma \sum_{x'} T(x'|x, u) V^*(x')$.

## Technical details

- Stochastic Dynamics and Assumptions: The state transition is defined by $x_{k+1} = f(x_k, u_k, w_k)$. The key assumption is that the disturbance $w_k$ must be Markovian, meaning its distribution depends only on $(x_k, u_k)$ and not on the process history.
- Bellman Recursion (Finite Horizon): The optimal cost-to-go $J_{k+1}(x_{k+1})$ is found recursively by minimizing the expected value of the sum of current and future costs: $\min_{u_k} \mathbb{E}[C(x_k, u_k, w_k) + J_{k+1}(x_{k+1})]$.
- Stochastic LQR Adaptation: For Linear Quadratic Regulator (LQR) problems with stochastic dynamics ($w$ is Gaussian, zero mean, covariance $\Sigma$), the optimal control policy remains linear feedback. The expectation of the quadratic cost term involving $w$ simplifies using the property: $\mathbb{E}[w^T P_{k+1} w] = \text{Trace}(P_{k+1} \Sigma)$.
- Optimal Action Selection (Learning Context): If the optimal Q-function $Q^*(x, u)$ is known (e.g., via an oracle), the optimal action $u^*$ can be found by solving a simple maximization problem: $u^* = \arg\max_{u} Q^*(x, u)$.

## Practical implications

- The MDP framework provides a rigorous mathematical tool for modeling and optimizing complex systems (e.g., autonomous vehicles, inventory management) where environmental uncertainty or disturbances are present.
- The Q-function formulation is critical in modern AI/ML applications (Reinforcement Learning), as it allows optimal action selection even when the underlying system dynamics (transition kernel $T$) are unknown or too complex to model explicitly.

## Topics

Stochastic Optimal Control, Markov Decision Processes (MDP), Dynamic Programming, Reinforcement Learning Theory, Linear Quadratic Regulator (LQR), Principles of Robot Autonomy (Companion Textbook), AA203 Optimal and Learning-Based Control Course Info

Source: https://www.youtube.com/watch?v=C9mLpI8Td9g