The Video Signal

Topic

Maximum Likelihood Estimation

All digests tagged Maximum Likelihood Estimation

· 1:16:30

Stanford CS229 Machine Learning | Spring 2026 | Lecture 9: K-Means and GMM (non-EM)

This lecture provides a deep dive into unsupervised machine learning algorithms, focusing on K-Means clustering and its probabilistic extension, Gaussian Mixture Models (GMM). The core mathematical framework for solving GMM is the Expectation-Maximization (EM) algorithm. Key concepts include understanding how to model structure without explicit labels, utilizing latent variables, and employing convex analysis via Jensen's inequality to derive the iterative estimation procedure.

Key takeaways

  1. Unsupervised vs. Supervised Learning 2:00

    Unlike supervised learning (where labels define separation), unsupervised learning aims to model inherent structure or clusters within unlabeled data, making it a fundamentally more challenging problem that requires stronger assumptions and accepting weaker guarantees.

  2. K-Means Clustering 5:50

    K-Means is an iterative algorithm where points are assigned to the nearest cluster center ($oldsymbol{ ext{mu}}_i$). The process involves two steps: (1) assigning each point to its closest $oldsymbol{ ext{mu}}$, and (2) recalculating the new cluster centers based on the arithmetic mean of all assigned points. Finding the optimal clustering is NP-hard, meaning initialization matters.

  3. Gaussian Mixture Models (GMM) 10:50

    GMMs are a probabilistic relaxation of K-Means, modeling data as mixtures of Gaussian distributions. Instead of hard assignments, points are assigned probabilities to belong to each source/cluster. The model is defined by means ($oldsymbol{ ext{mu}}$), covariances ($oldsymbol{ ext{Sigma}}$), and mixing proportions ($oldsymbol{f}$).

  4. Expectation-Maximization (EM) Algorithm 17:30

    The EM algorithm is used to estimate the parameters of latent variable models like GMM. It alternates between two steps: the E-step (calculating the probability $W_{ij}$ that each point belongs to each source given current parameter estimates) and the M-step (re-estimating all model parameters, including means and covariances, based on these probabilities).

  5. Convexity and Jensen's Inequality 30:00

    The EM algorithm relies on convex analysis. Convex functions are those where the line segment connecting any two points lies above the function graph (e.g., $x^2$). Jensen's inequality is used to derive a lower bound for the log-likelihood function, allowing the complex optimization problem to be solved iteratively.

Watch on YouTube Full article

· 1:02:14

Stanford CS229 Machine Learning | Spring 2026 | Lecture 3: Weighted Least Squares

This lecture provides a deep theoretical dive into classification and model building using the Maximum Likelihood Estimation (MLE) framework. The core principle demonstrated is that MLE allows generalizing modeling techniques from continuous regression (Least Squares) to discrete classification problems by introducing probabilistic interpretations. Key derivations include establishing Logistic Regression via the log-likelihood function, which transforms the complex product of probabilities into a numerically stable additive sum suitable for optimization using Stochastic Gradient Descent (SGD). Furthermore, the lecture compares various optimization methods—Gradient Descent, SGD, and Newton's Method—highlighting that while Newton's method is theoretically efficient, SGD remains the workhorse due to its scalability with massive datasets ($N$ and $D$).

Key takeaways

  1. Maximum Likelihood Principle (MLE) 1:35

    The MLE framework dictates setting up a probabilistic model (a forward model) of how data is generated. This principle generalizes across different problem types (continuous, discrete, etc.) by maximizing the likelihood function, which measures how probable the observed data is under a given set of parameters ($ heta$).

  2. Logistic Regression Derivation 5:50

    Classification problems are modeled using a link function (e.g., the sigmoid function) applied to a linear combination of features ($ ext{logit}(p)$). By applying MLE, the resulting log-likelihood function for binary classification leads directly to the standard form used in logistic regression.

  3. Optimization Method Comparison 17:50

    While Newton's method offers fast convergence and high precision, Stochastic Gradient Descent (SGD) is preferred in modern Machine Learning because it scales efficiently when dealing with massive datasets ($N$ and $D$), making it computationally feasible where full gradient computation is impossible.

Watch on YouTube Full article