# Stanford CS229 Machine Learning | Spring 2026 | Lecture 10: GMM (EM), PCA

## Executive summary

This lecture provides a deep dive into two foundational unsupervised machine learning techniques: Gaussian Mixture Models (GMM) solved via the Expectation-Maximization (EM) algorithm, and Principal Component Analysis (PCA). The discussion emphasizes the mathematical underpinnings of these methods. For GMM, the EM algorithm is shown to solve for latent variables by constructing a tractable lower bound using Jensen's inequality. For PCA, the technique identifies directions of maximum variance by performing Eigen Decomposition on the data's covariance matrix, highlighting the critical need for proper data centering and scaling.

## Key takeaways

- GMM via EM Algorithm: The Expectation-Maximization (EM) algorithm is used to estimate parameters ($ heta$) for GMM. It operates by constructing a lower bound $L( heta|T)$ of the log-likelihood, which allows optimization through alternating steps: E-step (calculating soft assignments/probabilities $Q_i(Z)$) and M-step (re-estimating parameters $oldsymbol{ heta}$).
- PCA Core Principle: PCA is a dimensionality reduction technique that finds orthogonal unit vectors (principal components, $U_k$) corresponding to the directions of maximal variance in the data. The process relies on finding the Eigen Decomposition ($oldsymbol{A} = oldsymbol{U} oldsymbol{ ext{diag}}(oldsymbol{ ext{eigenvalues}}) oldsymbol{U}^T$) of the covariance matrix.
- Mathematical Rigor (Jensen's Inequality): The EM algorithm leverages Jensen's inequality to transform an intractable expectation into a manageable lower bound, enabling iterative optimization. The selection of the conditional probability $Q(Z)$ is key to achieving this tight lower bound.

## Technical details

- Gaussian Mixture Models (GMM) & EM Algorithm: The GMM model assumes data points are generated from a mixture of Gaussian distributions. The EM algorithm iteratively refines parameters ($oldsymbol{ heta} = [oldsymbol{oldsymbol{ ext{mu}}}, oldsymbol{oldsymbol{ ext{Sigma}}}, oldsymbol{oldsymbol{ ext{phi}}}]$). The process involves: 1) E-step (Expectation): Calculating the soft assignment $Q_i(Z)$ for each point to every cluster. 2) M-step (Maximization): Re-estimating parameters by maximizing the lower bound surrogate function, which is derived using Jensen's inequality.
- Principal Component Analysis (PCA): PCA requires data to be centered ($oldsymbol{X} - oldsymbol{ ext{mu}}$) and often rescaled by variance to ensure all dimensions contribute equally. The principal components are the eigenvectors ($U_k$) of the covariance matrix, and their corresponding eigenvalues ($ ext{eigenvalues}$) represent the amount of captured variance. PCA is mathematically equivalent to performing Eigen Decomposition on the covariance matrix.
- Mathematical Constraints: The EM algorithm relies on the property that the lower bound $L( heta|T)$ must be monotonically increasing (tightness) and bounded above by the original log-likelihood. PCA requires the covariance matrix to be symmetric and positive semi-definite for Eigen Decomposition to hold.

## Practical implications

- Dimensionality reduction (PCA) is used to simplify high-dimensional data for visualization or feature engineering.
- GMM and EM are utilized in clustering tasks, providing a probabilistic interpretation of cluster assignments (soft assignment).
- Understanding the mathematical assumptions (e.g., latent variable structure, Gaussian assumption) is critical when deploying ML models.

## Topics

Gaussian Mixture Models (GMM), Expectation-Maximization (EM) Algorithm, Principal Component Analysis (PCA), Dimensionality Reduction, Latent Variables, Eigen Decomposition, Stanford Online, CS229 Machine Learning Course Website

Source: https://www.youtube.com/watch?v=sUS-eTa0l6s