Stanford CS229 Machine Learning | Spring 2026 | Lecture 3: Weighted Least Squares
Summary
This lecture provides a deep theoretical dive into classification and model building using the Maximum Likelihood Estimation (MLE) framework. The core principle demonstrated is that MLE allows generalizing modeling techniques from continuous regression (Least Squares) to discrete classification problems by introducing probabilistic interpretations. Key derivations include establishing Logistic Regression via the log-likelihood function, which transforms the complex product of probabilities into a numerically stable additive sum suitable for optimization using Stochastic Gradient Descent (SGD). Furthermore, the lecture compares various optimization methods—Gradient Descent, SGD, and Newton's Method—highlighting that while Newton's method is theoretically efficient, SGD remains the workhorse due to its scalability with massive datasets ($N$ and $D$).
Key takeaways
-
Maximum Likelihood Principle (MLE)
1:35
The MLE framework dictates setting up a probabilistic model (a forward model) of how data is generated. This principle generalizes across different problem types (continuous, discrete, etc.) by maximizing the likelihood function, which measures how probable the observed data is under a given set of parameters ($ heta$).
-
Logistic Regression Derivation
5:50
Classification problems are modeled using a link function (e.g., the sigmoid function) applied to a linear combination of features ($ ext{logit}(p)$). By applying MLE, the resulting log-likelihood function for binary classification leads directly to the standard form used in logistic regression.
-
Optimization Method Comparison
17:50
While Newton's method offers fast convergence and high precision, Stochastic Gradient Descent (SGD) is preferred in modern Machine Learning because it scales efficiently when dealing with massive datasets ($N$ and $D$), making it computationally feasible where full gradient computation is impossible.
Technical details
-
Probabilistic Modeling
51s
The assumption that the noise ($ ext{epsilon}_i$) in the data is Independent and Identically Distributed (IID) with a zero mean ($ ext{E}[ ext{epsilon}_i] = 0$) simplifies calculations significantly. The Gaussian distribution, $ ext{Normal}( ext{mean}, ext{variance})$, is frequently used because it can be fully characterized by its mean and variance.
-
Log-Likelihood Function
450s
The log-likelihood function ($ ext{L}( heta)$) converts the product of probabilities (which is numerically unstable) into a sum, making optimization feasible. For binary classification, this results in an additive cost function suitable for minimization.
-
Optimization Algorithms
1070s
The complexity comparison shows that while Newton's method has high theoretical efficiency, its computational cost per step is prohibitive ($O(N D^2 + D^3)$) compared to SGD or mini-batch gradient descent, which are optimized for large $N$ and $D$.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.