# Stanford CS229 Machine Learning | Spring 2026 | Lecture 3: Weighted Least Squares

## Executive summary

This lecture provides a deep theoretical dive into classification and model building using the Maximum Likelihood Estimation (MLE) framework. The core principle demonstrated is that MLE allows generalizing modeling techniques from continuous regression (Least Squares) to discrete classification problems by introducing probabilistic interpretations. Key derivations include establishing Logistic Regression via the log-likelihood function, which transforms the complex product of probabilities into a numerically stable additive sum suitable for optimization using Stochastic Gradient Descent (SGD). Furthermore, the lecture compares various optimization methods—Gradient Descent, SGD, and Newton's Method—highlighting that while Newton's method is theoretically efficient, SGD remains the workhorse due to its scalability with massive datasets ($N$ and $D$).

## Key takeaways

- Maximum Likelihood Principle (MLE): The MLE framework dictates setting up a probabilistic model (a forward model) of how data is generated. This principle generalizes across different problem types (continuous, discrete, etc.) by maximizing the likelihood function, which measures how probable the observed data is under a given set of parameters ($ heta$).
- Logistic Regression Derivation: Classification problems are modeled using a link function (e.g., the sigmoid function) applied to a linear combination of features ($ ext{logit}(p)$). By applying MLE, the resulting log-likelihood function for binary classification leads directly to the standard form used in logistic regression.
- Optimization Method Comparison: While Newton's method offers fast convergence and high precision, Stochastic Gradient Descent (SGD) is preferred in modern Machine Learning because it scales efficiently when dealing with massive datasets ($N$ and $D$), making it computationally feasible where full gradient computation is impossible.

## Technical details

- Probabilistic Modeling: The assumption that the noise ($ ext{epsilon}_i$) in the data is Independent and Identically Distributed (IID) with a zero mean ($ ext{E}[ ext{epsilon}_i] = 0$) simplifies calculations significantly. The Gaussian distribution, $ ext{Normal}( ext{mean}, ext{variance})$, is frequently used because it can be fully characterized by its mean and variance.
- Log-Likelihood Function: The log-likelihood function ($ ext{L}( heta)$) converts the product of probabilities (which is numerically unstable) into a sum, making optimization feasible. For binary classification, this results in an additive cost function suitable for minimization.
- Optimization Algorithms: The complexity comparison shows that while Newton's method has high theoretical efficiency, its computational cost per step is prohibitive ($O(N D^2 + D^3)$) compared to SGD or mini-batch gradient descent, which are optimized for large $N$ and $D$.

## Practical implications

- The MLE framework provides a generalized, principled approach for modeling diverse data types (e.g., count errors, discrete outcomes) beyond simple linear regression.
- Understanding the trade-offs between optimization methods (SGD vs. Newton's method) is crucial for building scalable ML systems that handle high dimensionality and large datasets.
- Logistic Regression remains a foundational model used in many modern AI components (like Softmax layers) despite the emergence of deep learning architectures.

## Topics

Maximum Likelihood Estimation, Classification Theory, Logistic Regression, Gaussian Distributions, Optimization Methods (SGD, Newton's Method), Link Functions, Stanford Online, CS229 Machine Learning Course Website

Source: https://www.youtube.com/watch?v=uJF_gL3jhxI