The Video Signal

Topic

Transformer Architecture

All digests tagged Transformer Architecture

Stanford CS229 Machine Learning | Spring 2026 | Lecture 16: Basic Concept in RL, Policy Gradient thumbnail

· 1:13:19

Stanford CS229 Machine Learning | Spring 2026 | Lecture 16: Basic Concept in RL, Policy Gradient

This lecture provides an advanced deep dive into optimizing Transformer architectures for efficiency and adapting Large Language Models (LLMs) for various downstream tasks. Key focus areas include reducing the quadratic complexity of attention through techniques like Grouped Query Attention (GQA) and Sliding Window Attention; scaling models using Mixture of Experts (MoE) to decouple memory from compute; and exploring prompt-based methods such as In-Context Learning, Few-Shot, and Zero-Shot learning for task adaptation without updating model parameters.

Key takeaways

  1. Efficiency in Attention Mechanisms 20:04

    The standard self-attention mechanism has $O(T^2)$ complexity (where T is sequence length). To mitigate this, techniques like Grouped Query Attention (GQA) reduce the number of keys and values used across heads by mapping multiple query groups to a smaller set of shared keys/values. Similarly, Sliding Window Attention limits attention to only recent history, reducing complexity to $O(T imes W)$ where W is the window size.

  2. Scaling with Mixture of Experts (MoE) 42:28

    MoE allows models to have a large total parameter count (e.g., 30B) while keeping the active computation small (e.g., 3B). This is achieved by using a routing module that directs an input vector to only a subset of specialized expert sub-networks, significantly improving compute efficiency.

  3. LLM Adaptation via Prompting 56:48

    For downstream tasks (e.g., sentiment analysis), models can be adapted using In-Context Learning (ICL). This involves concatenating task examples and the test input into the prompt sequence without updating model parameters, which is fundamentally different from traditional fine-tuning.

  4. Supervised Fine-Tuning (SFT) 1:04:00

    SFT involves collecting data in an instruction/answer pair format and training the model by minimizing the negative log likelihood of predicting the answer ($Y$) given the instruction ($X$). This is a supervised process that updates the model's weights.

Watch on YouTube Full article

Stanford CS229 Machine Learning | Spring 2026 | Lecture 14: Transformers, In-Context Learning thumbnail

· 1:17:32

Stanford CS229 Machine Learning | Spring 2026 | Lecture 14: Transformers, In-Context Learning

This lecture provides a deep technical dive into Large Language Models (LLMs), focusing on the Transformer architecture. Key concepts covered include subword tokenization (using techniques like Byte Pair Encoding - BPE) to convert text into numerical IDs, and modeling language distribution using auto-regressive conditional probabilities. The core mechanism is the self-attention layer, which uses Query (Q), Key (K), and Value (V) vectors to determine dependencies between tokens. A critical technical discussion centers on computational efficiency: standard attention has a quadratic $O(T^2)$ dependency on sequence length ($T$), leading to memory and computation bottlenecks for long contexts. Solutions like masking and specialized variants (e.g., Flash Attention) are discussed as methods to maintain performance while managing resource constraints.

Key takeaways

  1. Auto-Regressive Modeling 1:35

    LLMs model language distribution by decomposing the joint probability into a product of conditional probabilities: $P(X_t | X_{1...t-1})$. This requires generating tokens sequentially, making the process auto-regressive.

  2. Subword Tokenization (BPE) 3:30

    To handle rare or long words efficiently, models use subword tokenization (e.g., BPE), which breaks down text into smaller units (tokens). This allows leveraging common prefixes and suffixes to understand novel words.

  3. Self-Attention Mechanism 11:10

    The attention mechanism calculates the relevance between all tokens in a sequence using Query ($Q$), Key ($K$), and Value ($V$) vectors via an inner product: $ ext{Softmax}(Q K^T / ext{scale}) V$. The output is a weighted linear combination of $V$ vectors.

  4. Causal Masking for Auto-regression 21:20

    To ensure the model only predicts based on previous tokens (required for generation), a masking technique is applied to the attention matrix, setting all future dependencies to $- ext{infinity}$ before applying Softmax.

  5. Computational Bottleneck ($O(T^2)$) 30:00

    The standard self-attention mechanism has a computational and memory complexity that scales quadratically with sequence length $T$ (i.e., $O(T^2)$). This is the primary limitation for processing very long contexts.

Watch on YouTube Full article