Stanford CS229 Machine Learning | Spring 2026 | Lecture 4: Exponential Family, GLMs classification
Summary
The lecture introduces the Exponential Family as a unifying framework for numerous statistical models, including linear regression, logistic regression, Gaussian error modeling, and multinomial classification (Softmax). By showing that these diverse distributions share a common mathematical form, the theory demonstrates that core machine learning procedures—such as inference (calculating expected values) and parameter estimation (learning)—can be generalized across all members of this family. This foundational understanding is critical for modern AI architectures, particularly those utilizing Softmax in large language models.
Key takeaways
-
The Exponential Family Unification
2:00
Many common distributions (Bernoulli, Gaussian, Multinomial) can be written into a single functional form. This allows for the generalization of model building and analysis across different data types.
-
Inference and Learning are Generalized
3:30
The structure of the exponential family ensures that key statistical operations, such as computing the expected value (inference) and performing maximum likelihood estimation (learning), follow a consistent mathematical pattern regardless of the specific distribution used.
-
Softmax in Multiclass Classification
6:10
The Softmax function is presented as the standard mechanism for multiclass classification, allowing prediction over $K$ discrete values. Mathematically, it fits within the exponential family structure and generalizes logistic regression (the two-class case).
-
Generalized Linear Models (GLMs)
10:50
The GLM framework ties the abstract error models to practical supervised learning. The model predicts a distribution over $Y$ given features $X$, and the loss function is derived directly from maximizing the log-likelihood of this distribution.
Technical details
-
Exponential Family Form
160s
The probability distribution $P(Y|A)$ must conform to a specific functional form involving the sufficient statistic ($T_Y$), the natural parameter ($ heta$), and the log partition function ($ ext{log } Z( heta)$). This structure is key because it dictates how inference and learning are performed.
-
Multinomial Softmax
420s
For $K$ classes, the probability distribution is calculated by taking a weighted sum of one-hot vectors (the support). The final prediction uses the Softmax function to normalize raw scores ($ heta^T X$) into probabilities that sum to one.
-
Cross-Entropy Loss
750s
The loss function for classification is derived from maximizing the log-likelihood of the data. This results in a Cross-Entropy form, which unifies the training process across various GLM types.
-
Model Parameters vs. Canonical Parameters
1030s
It is crucial to distinguish between model parameters ($ heta$, what is learned) and canonical parameters (the natural parameter used in the exponential family formulation). The link function connects these two.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.