Stanford CS229 Machine Learning | Spring 2026 | Lecture 12: Representation Learning
The lecture provides a deep dive into advanced generative and adaptation techniques, covering Diffusion Models, Foundation Model paradigms, and Representation Learning. Key technical focus areas include deriving the loss function for diffusion models by comparing true and predicted mean reconstructions ($\mu_{ ilde{t}, t}$ vs $\mu_{ heta, t}$). For large language models (LLMs), the discussion highlights the shift toward using massive unlabelled data for pre-training and adapting via zero-shot or few-shot methods. Finally, it details advanced parameter efficiency techniques like Linear Probing and Low-Rank Adaptation (LoRA), which are crucial for managing memory and compute when deploying large models across multiple users.
Key takeaways
-
Diffusion Model Training Objective
33:55
The training objective is derived from the lower bound on $\log p_ heta(x_0)$, simplifying to minimizing a sum of terms ($\mathcal{L}_{t-1}$) that compares the mean predicted reconstruction ($\mu_{ heta, t}$) with the true process mean ($\mu_{ ilde{t}, t}$). This loss function is used to train the model to predict noise/reconstruct missing information at each time step.
-
Foundation Model Paradigm Shift
1:02:00
Modern LLMs utilize a paradigm shift involving pre-training on massive unlabelled data and adapting via zero-shot or few-shot learning, reducing the need for collecting extensive downstream labeled datasets. The goal is to build a general foundation that can be adapted to unlimited tasks.
-
Low-Rank Adaptation (LoRA) Efficiency
1:15:00
To adapt massive pre-trained models without changing all parameters ($\theta$), LoRA restricts the weight update matrix $\Delta W$ to a low rank ($A B$). This significantly reduces the number of trainable parameters and is critical for memory efficiency, especially when serving multiple users.
