Stanford CS229 Machine Learning | Spring 2026 | Lecture 10: GMM (EM), PCA
Summary
This lecture provides a deep dive into two foundational unsupervised machine learning techniques: Gaussian Mixture Models (GMM) solved via the Expectation-Maximization (EM) algorithm, and Principal Component Analysis (PCA). The discussion emphasizes the mathematical underpinnings of these methods. For GMM, the EM algorithm is shown to solve for latent variables by constructing a tractable lower bound using Jensen's inequality. For PCA, the technique identifies directions of maximum variance by performing Eigen Decomposition on the data's covariance matrix, highlighting the critical need for proper data centering and scaling.
Key takeaways
-
GMM via EM Algorithm
23:20
The Expectation-Maximization (EM) algorithm is used to estimate parameters ($ heta$) for GMM. It operates by constructing a lower bound $L( heta|T)$ of the log-likelihood, which allows optimization through alternating steps: E-step (calculating soft assignments/probabilities $Q_i(Z)$) and M-step (re-estimating parameters $oldsymbol{ heta}$).
-
PCA Core Principle
1:03:20
PCA is a dimensionality reduction technique that finds orthogonal unit vectors (principal components, $U_k$) corresponding to the directions of maximal variance in the data. The process relies on finding the Eigen Decomposition ($oldsymbol{A} = oldsymbol{U} oldsymbol{ ext{diag}}(oldsymbol{ ext{eigenvalues}}) oldsymbol{U}^T$) of the covariance matrix.
-
Mathematical Rigor (Jensen's Inequality)
27:10
The EM algorithm leverages Jensen's inequality to transform an intractable expectation into a manageable lower bound, enabling iterative optimization. The selection of the conditional probability $Q(Z)$ is key to achieving this tight lower bound.
Technical details
-
Gaussian Mixture Models (GMM) & EM Algorithm
300s
The GMM model assumes data points are generated from a mixture of Gaussian distributions. The EM algorithm iteratively refines parameters ($oldsymbol{ heta} = [oldsymbol{oldsymbol{ ext{mu}}}, oldsymbol{oldsymbol{ ext{Sigma}}}, oldsymbol{oldsymbol{ ext{phi}}}]$). The process involves: 1) E-step (Expectation): Calculating the soft assignment $Q_i(Z)$ for each point to every cluster. 2) M-step (Maximization): Re-estimating parameters by maximizing the lower bound surrogate function, which is derived using Jensen's inequality.
-
Principal Component Analysis (PCA)
3800s
PCA requires data to be centered ($oldsymbol{X} - oldsymbol{ ext{mu}}$) and often rescaled by variance to ensure all dimensions contribute equally. The principal components are the eigenvectors ($U_k$) of the covariance matrix, and their corresponding eigenvalues ($ ext{eigenvalues}$) represent the amount of captured variance. PCA is mathematically equivalent to performing Eigen Decomposition on the covariance matrix.
-
Mathematical Constraints
1350s
The EM algorithm relies on the property that the lower bound $L( heta|T)$ must be monotonically increasing (tightness) and bounded above by the original log-likelihood. PCA requires the covariance matrix to be symmetric and positive semi-definite for Eigen Decomposition to hold.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.