Stanford CS229 Machine Learning | Spring 2026 | Lecture 4: Exponential Family, GLMs classification
The lecture introduces the Exponential Family as a unifying framework for numerous statistical models, including linear regression, logistic regression, Gaussian error modeling, and multinomial classification (Softmax). By showing that these diverse distributions share a common mathematical form, the theory demonstrates that core machine learning procedures—such as inference (calculating expected values) and parameter estimation (learning)—can be generalized across all members of this family. This foundational understanding is critical for modern AI architectures, particularly those utilizing Softmax in large language models.
Key takeaways
-
The Exponential Family Unification
2:00
Many common distributions (Bernoulli, Gaussian, Multinomial) can be written into a single functional form. This allows for the generalization of model building and analysis across different data types.
-
Inference and Learning are Generalized
3:30
The structure of the exponential family ensures that key statistical operations, such as computing the expected value (inference) and performing maximum likelihood estimation (learning), follow a consistent mathematical pattern regardless of the specific distribution used.
-
Softmax in Multiclass Classification
6:10
The Softmax function is presented as the standard mechanism for multiclass classification, allowing prediction over $K$ discrete values. Mathematically, it fits within the exponential family structure and generalizes logistic regression (the two-class case).
-
Generalized Linear Models (GLMs)
10:50
The GLM framework ties the abstract error models to practical supervised learning. The model predicts a distribution over $Y$ given features $X$, and the loss function is derived directly from maximizing the log-likelihood of this distribution.