# Stanford CS229 Machine Learning | Spring 2026 | Lecture 14: Transformers, In-Context Learning

## Executive summary

This lecture provides a deep technical dive into Large Language Models (LLMs), focusing on the Transformer architecture. Key concepts covered include subword tokenization (using techniques like Byte Pair Encoding - BPE) to convert text into numerical IDs, and modeling language distribution using auto-regressive conditional probabilities. The core mechanism is the self-attention layer, which uses Query (Q), Key (K), and Value (V) vectors to determine dependencies between tokens. A critical technical discussion centers on computational efficiency: standard attention has a quadratic $O(T^2)$ dependency on sequence length ($T$), leading to memory and computation bottlenecks for long contexts. Solutions like masking and specialized variants (e.g., Flash Attention) are discussed as methods to maintain performance while managing resource constraints.

## Key takeaways

- Auto-Regressive Modeling: LLMs model language distribution by decomposing the joint probability into a product of conditional probabilities: $P(X_t | X_{1...t-1})$. This requires generating tokens sequentially, making the process auto-regressive.
- Subword Tokenization (BPE): To handle rare or long words efficiently, models use subword tokenization (e.g., BPE), which breaks down text into smaller units (tokens). This allows leveraging common prefixes and suffixes to understand novel words.
- Self-Attention Mechanism: The attention mechanism calculates the relevance between all tokens in a sequence using Query ($Q$), Key ($K$), and Value ($V$) vectors via an inner product: $ ext{Softmax}(Q K^T / ext{scale}) V$. The output is a weighted linear combination of $V$ vectors.
- Causal Masking for Auto-regression: To ensure the model only predicts based on previous tokens (required for generation), a masking technique is applied to the attention matrix, setting all future dependencies to $- ext{infinity}$ before applying Softmax.
- Computational Bottleneck ($O(T^2)$): The standard self-attention mechanism has a computational and memory complexity that scales quadratically with sequence length $T$ (i.e., $O(T^2)$). This is the primary limitation for processing very long contexts.

## Technical details

- Tokenization & Vocabulary: The process of converting text into a sequence of numerical IDs using a predefined vocabulary ($V$). Techniques like Byte Pair Encoding (BPE) are used to create subword units, which is more efficient than character or word-level tokenization.
- Probabilistic Modeling: The joint probability distribution $P(X_{1...T})$ is decomposed using the chain rule into conditional probabilities: $ ext{Softmax}(F_ heta(X_{0...t-1}))$. The loss function minimizes the negative log likelihood (NLL) over the sequence.
- Transformer Architecture: The model structure alternates between an Attention layer and a Multi-Layer Perceptron (MLP). Residual connections and normalization layers (like RMSNorm) are used to stabilize training.
- Attention Complexity & Optimization: Standard attention requires $O(T^2)$ time and memory complexity due to the calculation of the full $Q K^T$ matrix. To mitigate this, techniques like Flash Attention are used to reduce memory footprint by avoiding explicit storage of the large attention matrix.

## Practical implications

- Understanding the $O(T^2)$ dependency is crucial for designing systems that handle long context windows (e.g., RAG or document summarization).
- The concept of masking must be implemented correctly to enforce auto-regressive generation, ensuring the model cannot 'see' future tokens.
- When optimizing LLM inference, focus on memory efficiency and reducing complexity dependency on sequence length $T$ using optimized attention variants.

## Topics

Large Language Models (LLMs), Transformer Architecture, Self-Attention Mechanism, Auto-Regressive Generation, Byte Pair Encoding (BPE), Computational Complexity, Stanford CS229 Machine Learning, OpenAI Tokenizer Playground

Source: https://www.youtube.com/watch?v=pwQ0l4hFCVI