# Stanford CS229 Machine Learning | Spring 2026 | Lecture 12: Representation Learning

## Executive summary

The lecture provides a deep dive into advanced generative and adaptation techniques, covering Diffusion Models, Foundation Model paradigms, and Representation Learning. Key technical focus areas include deriving the loss function for diffusion models by comparing true and predicted mean reconstructions ($\mu_{ ilde{t}, t}$ vs $\mu_{ heta, t}$). For large language models (LLMs), the discussion highlights the shift toward using massive unlabelled data for pre-training and adapting via zero-shot or few-shot methods. Finally, it details advanced parameter efficiency techniques like Linear Probing and Low-Rank Adaptation (LoRA), which are crucial for managing memory and compute when deploying large models across multiple users.

## Key takeaways

- Diffusion Model Training Objective: The training objective is derived from the lower bound on $\log p_ heta(x_0)$, simplifying to minimizing a sum of terms ($\mathcal{L}_{t-1}$) that compares the mean predicted reconstruction ($\mu_{ heta, t}$) with the true process mean ($\mu_{ ilde{t}, t}$). This loss function is used to train the model to predict noise/reconstruct missing information at each time step.
- Foundation Model Paradigm Shift: Modern LLMs utilize a paradigm shift involving pre-training on massive unlabelled data and adapting via zero-shot or few-shot learning, reducing the need for collecting extensive downstream labeled datasets. The goal is to build a general foundation that can be adapted to unlimited tasks.
- Low-Rank Adaptation (LoRA) Efficiency: To adapt massive pre-trained models without changing all parameters ($\theta$), LoRA restricts the weight update matrix $\Delta W$ to a low rank ($A B$). This significantly reduces the number of trainable parameters and is critical for memory efficiency, especially when serving multiple users.

## Technical details

- Diffusion Model Loss Function: The loss function minimizes the difference between two Gaussian means: $\mathbb{E}_{Q} [ ||\mu_{ heta, t}(x_{t-1}|x_t, x_0) - \mu_{ ilde{t}, t}(x_{t-1}|x_t, x_0)||_2^2 ]$. In practice, this is often simplified to predicting the noise $\epsilon$ added at each step.
- Representation Learning (Linear Probing): This technique involves training a feature extractor $f_ heta(X)$ on massive data and then solving a downstream task using only a linear head ($\hat{W}$) applied to the fixed representation vector. This decouples feature extraction from task classification.
- Parameter Efficient Fine-Tuning (LoRA): Instead of optimizing all parameters $W$, LoRA approximates the update $\Delta W$ as a low-rank matrix product ($A B$). This drastically reduces trainable parameters and memory overhead, allowing multiple user adapters to share the base model weights.

## Practical implications

- Understanding the efficiency gains of LoRA is critical for deploying large models (LLMs) in resource-constrained or multi-tenant environments.
- The shift to zero-shot and few-shot learning minimizes data collection overhead, impacting system design requirements for downstream task adaptation.
- Diffusion model training requires careful management of stochastic processes and loss function optimization over time steps $t$.

## Topics

Generative Models, Diffusion Models, Foundation Models, Large Language Models (LLMs), Representation Learning, Low-Rank Adaptation (LoRA), Stanford CS229 Machine Learning, Stanford Online AI Programs

Source: https://www.youtube.com/watch?v=_kREM2UAiJ8