Stanford CME 295 Transformers & LLMs | Fall 2026 | Lecture 3 - LLM training
Summary
This lecture provides a comprehensive, technical overview of the entire Large Language Model (LLM) training lifecycle, moving beyond basic transformer architecture. The process is structured into distinct stages: Pre-training (learning general language patterns via next token prediction on massive corpora), Mid-training (scaling context length using techniques like RoPE frequency interpolation), Supervised Fine-tuning (SFT) (aligning behavior using high-quality instruction/response pairs), and finally, Alignment (using methods like RLHF, DPO, and specialized reasoning training). Key engineering concepts covered include parameter efficiency techniques like Low-Rank Adaptation (LoRA) and advanced training strategies like On-Policy Distillation (OPD).
Key takeaways
-
LLM Training is a Multi-Stage Pipeline
52:27
LLMs are not trained in one step. The process involves Pre-training (general knowledge), Mid-training (long context handling), SFT (behavior alignment), and Alignment (human preference/reasoning).
-
Efficiency through Low-Rank Adaptation (LoRA)
1:00:15
To fine-tune massive models without updating all parameters, LoRA decomposes the weight matrix into two much smaller, tunable matrices (B and A), drastically reducing computational and storage overhead.
-
Alignment Methods: DPO vs. RLHF
1:22:18
While Reinforcement Learning from Human Feedback (RLHF) is powerful but complex, Direct Preference Optimization (DPO) offers a supervised, mathematically simpler alternative for aligning the model based on pairwise human preferences.
-
Reasoning Requires Verifiable Rewards
To improve reasoning capabilities (e.g., math/coding), the model must be trained using verifiable rewards (ground truth) rather than general human preference, often utilizing algorithms like Group Relative Policy Optimization (GRPO).
-
On-Policy Distillation (OPD)
OPD combines the benefits of both dense, token-level signals (on-policy) and the stability of distillation, where a student model is trained against a teacher model's token-by-token predictions.
Technical details
-
Cross-Entropy Loss (Pre-training)
1033s
The core objective function for next token prediction. The goal is to find parameters $\theta$ that maximize the probability of the model seeing the training data, which is mathematically minimized by minimizing the negative log-likelihood (Cross-Entropy Loss).
-
Mixture of Experts (MoE)
1433s
An architecture that improves compute efficiency by selecting only specific 'experts' to activate for a given forward pass, placing this mechanism within the feed-forward neural network.
-
Long Context Training (RoPE)
2341s
To handle sequences much longer than the training context (e.g., 4K to 128K), techniques like RoPE frequency interpolation (e.g., YARN) are used. This tweaks the positional embeddings to make the model familiar with the required frequency ranges.
-
Supervised Fine-Tuning (SFT)
3147s
The process of tweaking model behavior using high-quality input/output pairs (e.g., instruction/response). The loss is localized, focusing only on predicting the desired output tokens, not the input prompt.
-
Direct Preference Optimization (DPO)
4938s
A supervised learning method for alignment that minimizes a loss function derived from the Bradley-Terry formulation, incentivizing the model to increase the probability of generating a winning answer relative to a losing answer.
Mentioned resources
- Stanford CME 295
- GLM5
- Artificial Intelligence Index
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.