# Stanford CS229 Machine Learning | Spring 2026 | Lecture 4: Exponential Family, GLMs classification

## Executive summary

The lecture introduces the Exponential Family as a unifying framework for numerous statistical models, including linear regression, logistic regression, Gaussian error modeling, and multinomial classification (Softmax). By showing that these diverse distributions share a common mathematical form, the theory demonstrates that core machine learning procedures—such as inference (calculating expected values) and parameter estimation (learning)—can be generalized across all members of this family. This foundational understanding is critical for modern AI architectures, particularly those utilizing Softmax in large language models.

## Key takeaways

- The Exponential Family Unification: Many common distributions (Bernoulli, Gaussian, Multinomial) can be written into a single functional form. This allows for the generalization of model building and analysis across different data types.
- Inference and Learning are Generalized: The structure of the exponential family ensures that key statistical operations, such as computing the expected value (inference) and performing maximum likelihood estimation (learning), follow a consistent mathematical pattern regardless of the specific distribution used.
- Softmax in Multiclass Classification: The Softmax function is presented as the standard mechanism for multiclass classification, allowing prediction over $K$ discrete values. Mathematically, it fits within the exponential family structure and generalizes logistic regression (the two-class case).
- Generalized Linear Models (GLMs): The GLM framework ties the abstract error models to practical supervised learning. The model predicts a distribution over $Y$ given features $X$, and the loss function is derived directly from maximizing the log-likelihood of this distribution.

## Technical details

- Exponential Family Form: The probability distribution $P(Y|A)$ must conform to a specific functional form involving the sufficient statistic ($T_Y$), the natural parameter ($ heta$), and the log partition function ($ ext{log } Z( heta)$). This structure is key because it dictates how inference and learning are performed.
- Multinomial Softmax: For $K$ classes, the probability distribution is calculated by taking a weighted sum of one-hot vectors (the support). The final prediction uses the Softmax function to normalize raw scores ($ heta^T X$) into probabilities that sum to one.
- Cross-Entropy Loss: The loss function for classification is derived from maximizing the log-likelihood of the data. This results in a Cross-Entropy form, which unifies the training process across various GLM types.
- Model Parameters vs. Canonical Parameters: It is crucial to distinguish between model parameters ($ heta$, what is learned) and canonical parameters (the natural parameter used in the exponential family formulation). The link function connects these two.

## Practical implications

- Understanding the exponential family provides a deep, foundational understanding of why modern AI models (like those using Softmax in LLMs) work, showing that they are mathematically consistent extensions of simpler statistical concepts.
- The framework dictates that model errors and loss functions can be derived from the underlying distribution type, simplifying the process of selecting appropriate training objectives.
- Knowledge of this theory is valuable for debugging or optimizing ML frameworks (like PyTorch/TensorFlow) because it explains the mathematical properties behind automatic differentiation and gradient descent.

## Topics

Exponential Family, Generalized Linear Models (GLMs), Softmax Activation, Multiclass Classification, Maximum Likelihood Estimation (MLE), Cross-Entropy Loss, Stanford CS229 Machine Learning, Stanford AI Programs Information

Source: https://www.youtube.com/watch?v=8gVi4Rk21Eg