Stanford AI Programs Information

Jul 31, 2026 · 1:16:30

Stanford CS229 Machine Learning | Spring 2026 | Lecture 9: K-Means and GMM (non-EM)

This lecture provides a deep dive into unsupervised machine learning algorithms, focusing on K-Means clustering and its probabilistic extension, Gaussian Mixture Models (GMM). The core mathematical framework for solving GMM is the Expectation-Maximization (EM) algorithm. Key concepts include understanding how to model structure without explicit labels, utilizing latent variables, and employing convex analysis via Jensen's inequality to derive the iterative estimation procedure.

Key takeaways

Unsupervised vs. Supervised Learning 2:00
Unlike supervised learning (where labels define separation), unsupervised learning aims to model inherent structure or clusters within unlabeled data, making it a fundamentally more challenging problem that requires stronger assumptions and accepting weaker guarantees.
K-Means Clustering 5:50
K-Means is an iterative algorithm where points are assigned to the nearest cluster center ($oldsymbol{ ext{mu}}_i$). The process involves two steps: (1) assigning each point to its closest $oldsymbol{ ext{mu}}$, and (2) recalculating the new cluster centers based on the arithmetic mean of all assigned points. Finding the optimal clustering is NP-hard, meaning initialization matters.
Gaussian Mixture Models (GMM) 10:50
GMMs are a probabilistic relaxation of K-Means, modeling data as mixtures of Gaussian distributions. Instead of hard assignments, points are assigned probabilities to belong to each source/cluster. The model is defined by means ($oldsymbol{ ext{mu}}$), covariances ($oldsymbol{ ext{Sigma}}$), and mixing proportions ($oldsymbol{f}$).
Expectation-Maximization (EM) Algorithm 17:30
The EM algorithm is used to estimate the parameters of latent variable models like GMM. It alternates between two steps: the E-step (calculating the probability $W_{ij}$ that each point belongs to each source given current parameter estimates) and the M-step (re-estimating all model parameters, including means and covariances, based on these probabilities).
Convexity and Jensen's Inequality 30:00
The EM algorithm relies on convex analysis. Convex functions are those where the line segment connecting any two points lies above the function graph (e.g., $x^2$). Jensen's inequality is used to derive a lower bound for the log-likelihood function, allowing the complex optimization problem to be solved iteratively.

Watch on YouTube Full article

Jul 30, 2026 · 1:02:13

Stanford CS229 Machine Learning | Spring 2026 | Lecture 8: Neural Networks 2 (Backprop)

This lecture provides a deep theoretical dive into Backpropagation and Automatic Differentiation, establishing it as an efficient method for computing gradients in complex neural networks. The core principle is that any differentiable network can be viewed as a 'differentiable circuit' or computational graph. This allows the gradient (the backward pass) to be computed with a time complexity proportional to the number of parameters ($O(N)$), matching the efficiency of the forward pass, regardless of how complex the underlying function is.

Key takeaways

Automatic Differentiation and Computational Graphs 2:00
The gradient computation (backward pass) can be viewed as traversing a differentiable circuit. This method allows for efficient calculation because it reuses intermediate results, unlike expanding the function into a traditional mathematical formula.
Efficiency of Gradient Computation 2:40
The fundamental theorem states that if a differentiable circuit of size $N$ computes a real-valued function, its gradient can also be computed in time complexity $O(N)$. This means the forward pass (evaluating loss) and the backward pass (calculating gradients) have similar computational costs.
Chain Rule Application for Backprop 5:50
Backpropagation is an application of the Chain Rule. By knowing the gradient with respect to an intermediate variable ($U$), one can compute the gradient with respect to its input ($Z$) using matrix multiplication involving the Jacobian (or transpose of the Jacobian). This process allows computation to proceed layer by layer.

Watch on YouTube Full article

Jul 29, 2026 · 1:14:12

Stanford CS229 Machine Learning | Spring 2026 | Lecture 4: Exponential Family, GLMs classification

The lecture introduces the Exponential Family as a unifying framework for numerous statistical models, including linear regression, logistic regression, Gaussian error modeling, and multinomial classification (Softmax). By showing that these diverse distributions share a common mathematical form, the theory demonstrates that core machine learning procedures—such as inference (calculating expected values) and parameter estimation (learning)—can be generalized across all members of this family. This foundational understanding is critical for modern AI architectures, particularly those utilizing Softmax in large language models.

Key takeaways

The Exponential Family Unification 2:00
Many common distributions (Bernoulli, Gaussian, Multinomial) can be written into a single functional form. This allows for the generalization of model building and analysis across different data types.
Inference and Learning are Generalized 3:30
The structure of the exponential family ensures that key statistical operations, such as computing the expected value (inference) and performing maximum likelihood estimation (learning), follow a consistent mathematical pattern regardless of the specific distribution used.
Softmax in Multiclass Classification 6:10
The Softmax function is presented as the standard mechanism for multiclass classification, allowing prediction over $K$ discrete values. Mathematically, it fits within the exponential family structure and generalizes logistic regression (the two-class case).
Generalized Linear Models (GLMs) 10:50
The GLM framework ties the abstract error models to practical supervised learning. The model predicts a distribution over $Y$ given features $X$, and the loss function is derived directly from maximizing the log-likelihood of this distribution.

Watch on YouTube Full article