Stanford CS229 Machine Learning | Spring 2026 | Lecture 12: Representation Learning
Summary
The lecture provides a deep dive into advanced generative and adaptation techniques, covering Diffusion Models, Foundation Model paradigms, and Representation Learning. Key technical focus areas include deriving the loss function for diffusion models by comparing true and predicted mean reconstructions ($\mu_{ ilde{t}, t}$ vs $\mu_{ heta, t}$). For large language models (LLMs), the discussion highlights the shift toward using massive unlabelled data for pre-training and adapting via zero-shot or few-shot methods. Finally, it details advanced parameter efficiency techniques like Linear Probing and Low-Rank Adaptation (LoRA), which are crucial for managing memory and compute when deploying large models across multiple users.
Key takeaways
-
Diffusion Model Training Objective
33:55
The training objective is derived from the lower bound on $\log p_ heta(x_0)$, simplifying to minimizing a sum of terms ($\mathcal{L}_{t-1}$) that compares the mean predicted reconstruction ($\mu_{ heta, t}$) with the true process mean ($\mu_{ ilde{t}, t}$). This loss function is used to train the model to predict noise/reconstruct missing information at each time step.
-
Foundation Model Paradigm Shift
1:02:00
Modern LLMs utilize a paradigm shift involving pre-training on massive unlabelled data and adapting via zero-shot or few-shot learning, reducing the need for collecting extensive downstream labeled datasets. The goal is to build a general foundation that can be adapted to unlimited tasks.
-
Low-Rank Adaptation (LoRA) Efficiency
1:15:00
To adapt massive pre-trained models without changing all parameters ($\theta$), LoRA restricts the weight update matrix $\Delta W$ to a low rank ($A B$). This significantly reduces the number of trainable parameters and is critical for memory efficiency, especially when serving multiple users.
Technical details
-
Diffusion Model Loss Function
2035s
The loss function minimizes the difference between two Gaussian means: $\mathbb{E}_{Q} [ ||\mu_{ heta, t}(x_{t-1}|x_t, x_0) - \mu_{ ilde{t}, t}(x_{t-1}|x_t, x_0)||_2^2 ]$. In practice, this is often simplified to predicting the noise $\epsilon$ added at each step.
-
Representation Learning (Linear Probing)
3100s
This technique involves training a feature extractor $f_ heta(X)$ on massive data and then solving a downstream task using only a linear head ($\hat{W}$) applied to the fixed representation vector. This decouples feature extraction from task classification.
-
Parameter Efficient Fine-Tuning (LoRA)
4500s
Instead of optimizing all parameters $W$, LoRA approximates the update $\Delta W$ as a low-rank matrix product ($A B$). This drastically reduces trainable parameters and memory overhead, allowing multiple user adapters to share the base model weights.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.