Stanford CS229 Machine Learning | Spring 2026 | Lecture 14: Transformers, In-Context Learning
Summary
This lecture provides a deep technical dive into Large Language Models (LLMs), focusing on the Transformer architecture. Key concepts covered include subword tokenization (using techniques like Byte Pair Encoding - BPE) to convert text into numerical IDs, and modeling language distribution using auto-regressive conditional probabilities. The core mechanism is the self-attention layer, which uses Query (Q), Key (K), and Value (V) vectors to determine dependencies between tokens. A critical technical discussion centers on computational efficiency: standard attention has a quadratic $O(T^2)$ dependency on sequence length ($T$), leading to memory and computation bottlenecks for long contexts. Solutions like masking and specialized variants (e.g., Flash Attention) are discussed as methods to maintain performance while managing resource constraints.
Key takeaways
-
Auto-Regressive Modeling
1:35
LLMs model language distribution by decomposing the joint probability into a product of conditional probabilities: $P(X_t | X_{1...t-1})$. This requires generating tokens sequentially, making the process auto-regressive.
-
Subword Tokenization (BPE)
3:30
To handle rare or long words efficiently, models use subword tokenization (e.g., BPE), which breaks down text into smaller units (tokens). This allows leveraging common prefixes and suffixes to understand novel words.
-
Self-Attention Mechanism
11:10
The attention mechanism calculates the relevance between all tokens in a sequence using Query ($Q$), Key ($K$), and Value ($V$) vectors via an inner product: $ ext{Softmax}(Q K^T / ext{scale}) V$. The output is a weighted linear combination of $V$ vectors.
-
Causal Masking for Auto-regression
21:20
To ensure the model only predicts based on previous tokens (required for generation), a masking technique is applied to the attention matrix, setting all future dependencies to $- ext{infinity}$ before applying Softmax.
-
Computational Bottleneck ($O(T^2)$)
30:00
The standard self-attention mechanism has a computational and memory complexity that scales quadratically with sequence length $T$ (i.e., $O(T^2)$). This is the primary limitation for processing very long contexts.
Technical details
-
Tokenization & Vocabulary
210s
The process of converting text into a sequence of numerical IDs using a predefined vocabulary ($V$). Techniques like Byte Pair Encoding (BPE) are used to create subword units, which is more efficient than character or word-level tokenization.
-
Probabilistic Modeling
350s
The joint probability distribution $P(X_{1...T})$ is decomposed using the chain rule into conditional probabilities: $ ext{Softmax}(F_ heta(X_{0...t-1}))$. The loss function minimizes the negative log likelihood (NLL) over the sequence.
-
Transformer Architecture
1600s
The model structure alternates between an Attention layer and a Multi-Layer Perceptron (MLP). Residual connections and normalization layers (like RMSNorm) are used to stabilize training.
-
Attention Complexity & Optimization
1800s
Standard attention requires $O(T^2)$ time and memory complexity due to the calculation of the full $Q K^T$ matrix. To mitigate this, techniques like Flash Attention are used to reduce memory footprint by avoiding explicit storage of the large attention matrix.
Mentioned resources
- Stanford CS229 Machine Learning
- OpenAI Tokenizer Playground
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.