Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 1 - Transformers
This lecture provides a deep dive into the mechanics of Transformers and Large Language Models (LLMs), tracing the evolution of NLP from early models like RNNs to modern, scalable architectures. The core focus is on the self-attention mechanism, which allows tokens to compute representations based on all other tokens in the sequence. The lecture details the structure of the Transformer (Encoder and Decoder), the role of specialized tokens (e.g., `[BOS]`, `[EOS]`), and critical engineering components like position embeddings, residual connections, and multi-head attention.
Key takeaways
-
The Transformer Architecture
18:13
The Transformer, introduced in 2017, replaced recurrent models (RNNs) by relying entirely on attention mechanisms, enabling massive scalability and parallel processing. It consists of an Encoder (processes source text) and a Decoder (generates the target sequence).
-
Self-Attention Mechanism
22:13
The representation of a token is computed as a weighted average of all other tokens, determined by the Query (Q), Key (K), and Value (V) vectors. The core calculation involves the softmax of (Q * K^T) / $\sqrt{d_k}$ multiplied by V.
-
Tokenization Techniques
3:36
Text must be converted to numbers. Subword tokenization, specifically Byte Pair Encoding (BPE), is the most popular method as it balances vocabulary size and sequence length, improving robustness and efficiency.
-
Model Enhancements
26:13
Modern implementations use techniques like residual connections (to aid gradient flow in deep networks), layer normalization (for stable convergence), and masked self-attention (in the decoder) to ensure causal generation.