Stanford Online

AI is moving past simple Q&A (quick, short, transactional) toward complex, long-running agentic tasks. OpenAI internal data suggests agents are now performing work across finance, recruiting, and legal that can take up to 8 hours of human effort [0:14:25].

For large enterprises (like United Health Group), long-form agentic work requires establishing strong governance, guardrails, and security protocols. Simply calling an API a 'super agent' is insufficient; true agency requires reasoning and decision-making capabilities [0:23:15].

Enterprises must manage the risks of vendor lock-in when restructuring workflows around a single model or API. Concerns include escalating token costs and geopolitical instability, making architectural flexibility paramount [0:47:28].

The most impactful approach is not to automate existing tasks (silos) but to fundamentally reimagine the entire end-to-end workflow—from patient intake to payment processing—given the new technological capabilities [0:35:12].

Successful adoption requires AI to be elevated from an IT or innovation problem to a CEO-level, cross-functional business mandate. Leadership must obsess over defining the core metrics (the 'what is the metric?') and driving change at scale [0:57:12].

Policy optimization methods (like TRPO and PPO) define a surrogate objective function to estimate the policy gradient, enabling continuous updates. PPO uses a clipped ratio ($ ext{clip}(r_{ heta}, 1- ext{eps}, 1+ ext{eps})$) to constrain the new policy's divergence from the old one, stabilizing training without requiring complex second-order optimization.

The basic model-based recipe (collect data $ ightarrow$ fit dynamics $P(s'|s, a)$ $ ightarrow$ plan) fails when dealing with complex or nonlinear dynamics because extrapolation outside the observed state distribution is unreliable. This issue of generalization and distribution shift must be addressed.

To improve model-based planning, uncertainty quantification is necessary. The distinction between Aleatoric (inherent noise) and Epistemic (model uncertainty) is crucial. Bayesian approaches treat this by modeling the posterior distribution over parameters ($ heta$), allowing for prediction averaging across all plausible models.

A practical approach to estimate model uncertainty is using ensembles: training multiple independent neural network copies. The average of their predictions approximates the predictive posterior distribution, effectively exploring multimodal solution landscapes without requiring complex analytical derivations.

Monte Carlo methods estimate the expected return ($G_t$) by rolling out an episode until a terminal state, requiring full episodes. Temporal Difference (TD) learning improves upon this by using bootstrapping—defining the target as the instantaneous reward plus the discounted future value ($ ext{Reward} + ext{Discounted Future Value}$), allowing for online updates and handling non-terminal environments.

SARSA is an on-policy algorithm, meaning it improves the policy ($ ext{e.g., } ext{epsilon-greedy}$) that is actively used to generate data in the environment. Q-learning is off-policy; it learns about a target optimal policy (the greedy policy) while using data generated by a different behavior policy (also $ ext{epsilon-greedy}$), which is crucial for utilizing historical or simulated data.

To overcome the curse of dimensionality inherent in tabular value function representations, RL methods transition to parametric functions (e.g., neural networks) that approximate $V(s)$ or $Q(s, a)$. This allows generalization across states and controls.

Deep Q Networks (DQN) stabilize learning using two methods: Experience Replay (storing transitions in a buffer to decorrelate samples, satisfying the IID assumption required for regression) and Fixed Q Targets (using a delayed copy of the network parameters ($ ext{Q}_{ ext{target}}$) to prevent the target from being a moving variable during optimization).

Avoidance sets and reachable sets are computed by solving a differential game using the Hamilton-Jacobi-Isaacs (HJI) equation. This involves reframing the Boolean problem of set membership into an optimal control cost function $h(x)$ [0:35].

To ensure safety over the entire trajectory, not just the endpoint, one must compute a Backward Reachable Tube (BRT). This is achieved by modifying the cost function to minimize the minimum value of $h(x)$ across the entire optimization horizon [7:40].

MPC solves an open-loop optimal control problem over a finite prediction horizon $[t, t+N_p]$ at each time step $t$. It then uses only the first computed input ($u_t$) and discards the rest of the plan, recomputing everything from scratch based on new state measurements (receding horizon) [19:30].

The primary goals when designing an MPC controller are ensuring persistent feasibility (the problem remains solvable at all future times) and guaranteeing stability (convergence to the desired state, e.g., the origin) [36:00].

To guarantee persistent feasibility in MPC, one must identify a control invariant set $\mathcal{C}$. This is a set where, if the system starts within it, there exists a control input $u$ that guarantees the next state remains inside $\mathcal{C}$ [38:00].

The state update is modeled as $x_{k+1} = f(x_k, u_k, w_k)$, where $w_k$ is a random disturbance. The system must adhere to the Markovian assumption: the probability distribution of $w_k$ can only depend on the current state ($x_k$) and control ($u_k$), not on the history of previous states or disturbances.

The notion of optimality is defined by minimizing the expected cost, $\mathbb{E}[ ext{Cost}]$, over all possible disturbance realizations. The solution uses a backward dynamic programming recursion (Bellman equation) to find the optimal closed-loop policy $\pi^*$.

To solve problems over an infinite number of stages, a discount factor ($\gamma \in [0, 1]$) is introduced to ensure the convergence of the expected cumulative reward. The optimal value function $V^*$ satisfies a fixed-point equation: $V^*(x) = \max_{u} \{ R(x, u) + \gamma \mathbb{E}[V^*(x')]\}$.

For computational tractability in learning settings where the transition kernel is unknown, the problem can be reformulated using the Q-function ($Q^*$), which represents the expected cumulative reward starting at state $x$ and taking action $u$, followed by optimal actions: $Q^*(x, u) = R(x, u) + \gamma \sum_{x'} T(x'|x, u) V^*(x')$.

While LQR is fundamentally designed to drive a state to the origin (regulation), it can be generalized to perform trajectory tracking by defining an auxiliary problem based on deviation variables. The optimal control law structure remains consistent: $u = u_{nominal} + ext{feedback term}$.

For nonlinear dynamics ($x_{k+1} = f(x_k, u_k)$), the tracking problem can be linearized by performing a Taylor expansion around the nominal trajectory ($\bar{x}, \bar{u}$), allowing the use of LQR techniques on the deviation variables.

Both iLQR and DDP are methods for solving nonlinear optimal control problems iteratively. iLQR linearizes dynamics and quadratizes costs, while DDP directly approximates the Bellman equation by quadratizing the Q-function, making it a second-order algorithm in terms of dynamic derivatives.

The ReAct framework combines Chain-of-Thought (reasoning) with tool-calling actions, creating an interleaved loop (thought $ ightarrow$ act $ ightarrow$ observe). This allows models to ground their reasoning in external knowledge sources, improving performance on tasks like HotpotQA and WebShop. [0:03]

RLEF is an end-to-end RL fine-tuning framework for coding agents that uses execution feedback (test pass/fail) as a binary reward signal. It employs a two-tier test strategy, using public tests for inference-time guidance and private tests for training the policy model via PPO. [0:12]

This method improves LLMs by having them critique their own outputs against a set of human-written principles (the 'Constitution'). This generates feedback for training the model's preference model, reducing reliance on constant human labeling. [0:15]

Model performance improves significantly by scaling up parameters, compute (compute on the x-axis), or dataset size. This exponential growth has been foundational for models like GPT-3 and PaLM.

Modern LLM applications are moving from single-turn chat formats to complex agentic workflows (e.g., Deep Research, Cloud Code). These systems can take a goal, plan steps, interact with environments/tools, and correct their actions until the goal is achieved.

In agentic systems, verifiers (like unit tests for code) are crucial feedback mechanisms. They allow models to verify generated outputs and guide self-correction, which is essential in verifiable domains like math and coding.

The most advanced frontier involves combining test time scaling (e.g., repeated sampling) with fine-tuning to generate synthetic, high-quality data. This creates a self-improving loop where the model trains on its own generated solutions.

LLM capability can be significantly enhanced at inference time by increasing compute (e.g., repeated sampling) without modifying model parameters or requiring fine-tuning, offering a new paradigm compared to traditional pre-training and fine-tuning.

By repeatedly querying the same problem (e.g., using Llama 3-8b or DeepSeek), selecting the correct response among candidates, models can achieve performance comparable to larger proprietary models like GPT-4o.

For repeated sampling to be effective, automated verification is crucial. The 'generation-verification gap' describes the large difference between the best possible outcome (Oracle selection) and what can be achieved using simple methods like majority voting.

The Arkon framework treats inference scaling as an architecture design problem, optimizing the combination of techniques (e.g., Fusion, Critic, Ranker) to maximize accuracy given a limited compute budget.

While outcome-based reward models (ORM) only check the final answer's correctness, process-based reward models (PRM) assign rewards per step of reasoning. PRMs are superior because they manage false positives better and encourage interpretable, human-endorsed steps [2:36:00].

The Weaver approach combines multiple weak verifiers (e.g., LLM judges, reward models) using techniques like Naive Bayes or logistic regression to create a single, highly capable verifier. This method significantly improves performance by leveraging the collective signal of diverse sources [3:42:00].

PRMs are shown to be more data-efficient than ORMs. Furthermore, the lecture demonstrates that scaling verification by increasing the number of verifiers (rather than just sampling more completions) can improve results while maintaining computational efficiency [3:42:00].

Advanced techniques involve using the model itself to generate data (e.g., Math-Shepherd) and then training a PRM on this synthetic, semi-automated data, allowing the system to self-improve its reasoning capabilities [2:56:00].

While test-time scaling (inference-based techniques like majority voting) improves accuracy by sampling outputs, train-time scaling uses the model's own filtered outputs to fine-tune and improve the model weights directly, creating a powerful closed feedback loop.

The STaR method bootstraps reasoning by generating solutions on a small set of examples. It filters for correct answers and then generates rationales (hints) for incorrect attempts, allowing the model to learn from failed paths iteratively.

DeepSeekMath introduced Group Relative Policy Optimization (GRPO), an efficient alternative to PPO that reduces memory overhead by using a group baseline instead of maintaining multiple policy copies, enabling scaling RL to larger models.

DAPO addresses training instability in long chain-of-thought reasoning by implementing asymmetric clipping (allowing bigger increases) and dynamic sampling (filtering out zero or one reward groups to maintain a useful gradient signal).

Studies show that titles and labels receive long fixations during encoding and are the most likely elements to be recalled, suggesting language significantly impacts how data visualizations are understood. The speaker notes this was historically under-explored in the visualization community.

Research suggests that 'more text is better' for general information displays, provided the text is relevant and properly annotated. This finding challenges traditional minimalist principles in UI/UX design.

Multimodal LLMs (MLLMs) process text and visuals by transforming inputs into embeddings. The 'cross-modality embedding architecture' uses a cross-attention mechanism, allowing information to flow between the two distinct sequences (e.g., image tokens interacting with text tokens).

The speaker notes that understanding how humans integrate text and visuals is complex, citing conflicting evidence across theories (Dual Coding vs. Cognitive Load Theory). The choice of representation depends heavily on the specific task or cognitive ability.

The policy gradient estimator is necessary because the dependency on parameters ($ heta$) is complex. The fundamental property that $ abla_{ heta} ext{E}_{ ext{P}_{ heta}}[ abla_{ heta} ext{log } ext{P}_{ heta}(a|s)]$ equals zero shows that without a reward function, there are no preferences to optimize for.

PPO is designed to stabilize RL training by using importance sampling and clipping the objective function. This prevents the new policy ($ heta$) from deviating too far from the old policy ($ heta_{old}$), which helps maintain stable learning.

The LLM generation process is modeled as a Markov Decision Process (MDP). The state ($s_t$) includes the history, and the action ($a_t$) is the next generated token. The reward function is typically applied only at the end of the trajectory based on whether the final answer matches the ground truth.

To train models for complex reasoning, RL can be used to reward the entire trajectory based on the final answer's correctness. This approach bypasses the need for explicit labeling of the internal 'thinking tokens,' focusing only on verifiable outcomes.

RL addresses sequential decision-making where actions have long-term ramifications. It requires balancing the trade-off between exploitation (using current best knowledge) and exploration (gathering information). Learning relies on maximizing a scalar reward signal rather than explicit labels or supervision.

An MDP formally describes an environment using five components: State Set ($S$), Action Set ($A$), Transition Dynamics ($P(s'|s, a)$), Reward Function ($R$), and Discount Factor ($\gamma$). The Markov property ensures that the future state transition depends only on the current state and action, not on history.

The value function $V^{\pi}(s)$ estimates the expected total payoff starting at state $s$ under policy $\pi$. The optimal value, $V^*(s)$, is the maximum possible return. These values are solved recursively using the Bellman equation, which relates the current state's value to the expected discounted future rewards.

The Policy Gradient algorithm optimizes a stochastic policy $\pi_{\theta}(a|s)$ by maximizing the expected return $E[R]$. The gradient is computed using the log-probability trick, allowing the calculation of $\nabla_{\theta} E[R]$ through sampling, even when the dependency on $\theta$ only affects the sampling distribution.

The standard self-attention mechanism has $O(T^2)$ complexity (where T is sequence length). To mitigate this, techniques like Grouped Query Attention (GQA) reduce the number of keys and values used across heads by mapping multiple query groups to a smaller set of shared keys/values. Similarly, Sliding Window Attention limits attention to only recent history, reducing complexity to $O(T imes W)$ where W is the window size.

MoE allows models to have a large total parameter count (e.g., 30B) while keeping the active computation small (e.g., 3B). This is achieved by using a routing module that directs an input vector to only a subset of specialized expert sub-networks, significantly improving compute efficiency.

For downstream tasks (e.g., sentiment analysis), models can be adapted using In-Context Learning (ICL). This involves concatenating task examples and the test input into the prompt sequence without updating model parameters, which is fundamentally different from traditional fine-tuning.

SFT involves collecting data in an instruction/answer pair format and training the model by minimizing the negative log likelihood of predicting the answer ($Y$) given the instruction ($X$). This is a supervised process that updates the model's weights.

LLMs model language distribution by decomposing the joint probability into a product of conditional probabilities: $P(X_t | X_{1...t-1})$. This requires generating tokens sequentially, making the process auto-regressive.

To handle rare or long words efficiently, models use subword tokenization (e.g., BPE), which breaks down text into smaller units (tokens). This allows leveraging common prefixes and suffixes to understand novel words.

The attention mechanism calculates the relevance between all tokens in a sequence using Query ($Q$), Key ($K$), and Value ($V$) vectors via an inner product: $ ext{Softmax}(Q K^T / ext{scale}) V$. The output is a weighted linear combination of $V$ vectors.

To ensure the model only predicts based on previous tokens (required for generation), a masking technique is applied to the attention matrix, setting all future dependencies to $- ext{infinity}$ before applying Softmax.

The standard self-attention mechanism has a computational and memory complexity that scales quadratically with sequence length $T$ (i.e., $O(T^2)$). This is the primary limitation for processing very long contexts.

The training objective is derived from the lower bound on $\log p_ heta(x_0)$, simplifying to minimizing a sum of terms ($\mathcal{L}_{t-1}$) that compares the mean predicted reconstruction ($\mu_{ heta, t}$) with the true process mean ($\mu_{ ilde{t}, t}$). This loss function is used to train the model to predict noise/reconstruct missing information at each time step.

Modern LLMs utilize a paradigm shift involving pre-training on massive unlabelled data and adapting via zero-shot or few-shot learning, reducing the need for collecting extensive downstream labeled datasets. The goal is to build a general foundation that can be adapted to unlimited tasks.

To adapt massive pre-trained models without changing all parameters ($\theta$), LoRA restricts the weight update matrix $\Delta W$ to a low rank ($A B$). This significantly reduces the number of trainable parameters and is critical for memory efficiency, especially when serving multiple users.

These models are generative frameworks that can be applied to images, videos, and robotics actions. They are noted as a superior approach compared to GANs and Variational Autoencoders (VAEs) for image generation.

The forward process is a fixed, known noising process that transforms $X_0$ to noise $X_T$. The reverse process is the learned component, parameterized by $ heta$, which predicts $p_ heta(X_{t-1}|X_t)$—the step-by-step denoising path.

The goal is to maximize the marginal likelihood $ ext{log } p_ heta(X_0)$ by applying the Evidence Lower Bound (ELBO). This involves minimizing the KL divergence between the true posterior $q(X_{t-1}|X_t, X_0)$ and the parameterized model's distribution.

The Expectation-Maximization (EM) algorithm is used to estimate parameters ($ heta$) for GMM. It operates by constructing a lower bound $L( heta|T)$ of the log-likelihood, which allows optimization through alternating steps: E-step (calculating soft assignments/probabilities $Q_i(Z)$) and M-step (re-estimating parameters $oldsymbol{ heta}$).

PCA is a dimensionality reduction technique that finds orthogonal unit vectors (principal components, $U_k$) corresponding to the directions of maximal variance in the data. The process relies on finding the Eigen Decomposition ($oldsymbol{A} = oldsymbol{U} oldsymbol{ ext{diag}}(oldsymbol{ ext{eigenvalues}}) oldsymbol{U}^T$) of the covariance matrix.

The EM algorithm leverages Jensen's inequality to transform an intractable expectation into a manageable lower bound, enabling iterative optimization. The selection of the conditional probability $Q(Z)$ is key to achieving this tight lower bound.

Unlike supervised learning (where labels define separation), unsupervised learning aims to model inherent structure or clusters within unlabeled data, making it a fundamentally more challenging problem that requires stronger assumptions and accepting weaker guarantees.

K-Means is an iterative algorithm where points are assigned to the nearest cluster center ($oldsymbol{ ext{mu}}_i$). The process involves two steps: (1) assigning each point to its closest $oldsymbol{ ext{mu}}$, and (2) recalculating the new cluster centers based on the arithmetic mean of all assigned points. Finding the optimal clustering is NP-hard, meaning initialization matters.

GMMs are a probabilistic relaxation of K-Means, modeling data as mixtures of Gaussian distributions. Instead of hard assignments, points are assigned probabilities to belong to each source/cluster. The model is defined by means ($oldsymbol{ ext{mu}}$), covariances ($oldsymbol{ ext{Sigma}}$), and mixing proportions ($oldsymbol{f}$).

The EM algorithm is used to estimate the parameters of latent variable models like GMM. It alternates between two steps: the E-step (calculating the probability $W_{ij}$ that each point belongs to each source given current parameter estimates) and the M-step (re-estimating all model parameters, including means and covariances, based on these probabilities).

The EM algorithm relies on convex analysis. Convex functions are those where the line segment connecting any two points lies above the function graph (e.g., $x^2$). Jensen's inequality is used to derive a lower bound for the log-likelihood function, allowing the complex optimization problem to be solved iteratively.