# Stanford CME 295 Transformers & LLMs | Fall 2026 | Lecture 3 - LLM training

## Executive summary

This lecture provides a comprehensive, technical overview of the entire Large Language Model (LLM) training lifecycle, moving beyond basic transformer architecture. The process is structured into distinct stages: Pre-training (learning general language patterns via next token prediction on massive corpora), Mid-training (scaling context length using techniques like RoPE frequency interpolation), Supervised Fine-tuning (SFT) (aligning behavior using high-quality instruction/response pairs), and finally, Alignment (using methods like RLHF, DPO, and specialized reasoning training). Key engineering concepts covered include parameter efficiency techniques like Low-Rank Adaptation (LoRA) and advanced training strategies like On-Policy Distillation (OPD).

## Key takeaways

- LLM Training is a Multi-Stage Pipeline: LLMs are not trained in one step. The process involves Pre-training (general knowledge), Mid-training (long context handling), SFT (behavior alignment), and Alignment (human preference/reasoning).
- Efficiency through Low-Rank Adaptation (LoRA): To fine-tune massive models without updating all parameters, LoRA decomposes the weight matrix into two much smaller, tunable matrices (B and A), drastically reducing computational and storage overhead.
- Alignment Methods: DPO vs. RLHF: While Reinforcement Learning from Human Feedback (RLHF) is powerful but complex, Direct Preference Optimization (DPO) offers a supervised, mathematically simpler alternative for aligning the model based on pairwise human preferences.
- Reasoning Requires Verifiable Rewards: To improve reasoning capabilities (e.g., math/coding), the model must be trained using verifiable rewards (ground truth) rather than general human preference, often utilizing algorithms like Group Relative Policy Optimization (GRPO).
- On-Policy Distillation (OPD): OPD combines the benefits of both dense, token-level signals (on-policy) and the stability of distillation, where a student model is trained against a teacher model's token-by-token predictions.

## Technical details

- Cross-Entropy Loss (Pre-training): The core objective function for next token prediction. The goal is to find parameters $\theta$ that maximize the probability of the model seeing the training data, which is mathematically minimized by minimizing the negative log-likelihood (Cross-Entropy Loss).
- Mixture of Experts (MoE): An architecture that improves compute efficiency by selecting only specific 'experts' to activate for a given forward pass, placing this mechanism within the feed-forward neural network.
- Long Context Training (RoPE): To handle sequences much longer than the training context (e.g., 4K to 128K), techniques like RoPE frequency interpolation (e.g., YARN) are used. This tweaks the positional embeddings to make the model familiar with the required frequency ranges.
- Supervised Fine-Tuning (SFT): The process of tweaking model behavior using high-quality input/output pairs (e.g., instruction/response). The loss is localized, focusing only on predicting the desired output tokens, not the input prompt.
- Direct Preference Optimization (DPO): A supervised learning method for alignment that minimizes a loss function derived from the Bradley-Terry formulation, incentivizing the model to increase the probability of generating a winning answer relative to a losing answer.

## Practical implications

- Understanding the multi-stage training pipeline (Pre-training $\rightarrow$ SFT $\rightarrow$ Alignment) is crucial for designing custom LLM solutions.
- Using parameter-efficient fine-tuning methods like LoRA allows organizations to adapt large, pre-trained models to specific use cases (e.g., spam detection, sentiment extraction) without massive computational overhead.
- The shift from general next-token prediction to localized loss functions (SFT) is necessary to guide the model's behavior toward specific, useful outputs (e.g., answering questions rather than completing text).
- For building reasoning agents, the focus must shift from general language knowledge to verifiable, ground-truth-based rewards, enabling the model to 'think' through steps.

## Topics

Large Language Models (LLMs), Transformer Architecture, Machine Learning Training, Model Alignment, Parameter Efficiency, Stanford CME 295, GLM5, Artificial Intelligence Index

Source: https://www.youtube.com/watch?v=S3bCDcJhz4s