Stanford CS229 Machine Learning | Spring 2026 | Lecture 16: Basic Concept in RL, Policy Gradient
Summary
This lecture provides an advanced deep dive into optimizing Transformer architectures for efficiency and adapting Large Language Models (LLMs) for various downstream tasks. Key focus areas include reducing the quadratic complexity of attention through techniques like Grouped Query Attention (GQA) and Sliding Window Attention; scaling models using Mixture of Experts (MoE) to decouple memory from compute; and exploring prompt-based methods such as In-Context Learning, Few-Shot, and Zero-Shot learning for task adaptation without updating model parameters.
Key takeaways
-
Efficiency in Attention Mechanisms
20:04
The standard self-attention mechanism has $O(T^2)$ complexity (where T is sequence length). To mitigate this, techniques like Grouped Query Attention (GQA) reduce the number of keys and values used across heads by mapping multiple query groups to a smaller set of shared keys/values. Similarly, Sliding Window Attention limits attention to only recent history, reducing complexity to $O(T imes W)$ where W is the window size.
-
Scaling with Mixture of Experts (MoE)
42:28
MoE allows models to have a large total parameter count (e.g., 30B) while keeping the active computation small (e.g., 3B). This is achieved by using a routing module that directs an input vector to only a subset of specialized expert sub-networks, significantly improving compute efficiency.
-
LLM Adaptation via Prompting
56:48
For downstream tasks (e.g., sentiment analysis), models can be adapted using In-Context Learning (ICL). This involves concatenating task examples and the test input into the prompt sequence without updating model parameters, which is fundamentally different from traditional fine-tuning.
-
Supervised Fine-Tuning (SFT)
1:04:00
SFT involves collecting data in an instruction/answer pair format and training the model by minimizing the negative log likelihood of predicting the answer ($Y$) given the instruction ($X$). This is a supervised process that updates the model's weights.
Technical details
-
Transformer Attention (QKV)
302s
Attention relies on Query (Q), Key (K), and Value (V) vectors. The core calculation involves $Q K^T / ext{scale} + ext{mask}$ followed by Softmax, then multiplication by V. For multiple heads, the process is replicated with different weights.
-
KV Cache and Memory Bottleneck
1502s
During generation (inference), only K and V need to be stored (the KV cache). This memory requirement limits the maximum batch size, often causing the GPU to become memory-bound rather than compute-bound.
-
Grouped Query Attention (GQA)
1830s
GQA reduces redundancy by having multiple query heads ($N_h$) map to a smaller number of shared key and value groups ($N_g$), where $N_g = N_h / ext{tall}$. This significantly reduces the memory footprint required for the KV cache.
-
Mixture of Experts (MoE) Routing
2600s
The routing module determines a subset $S$ of experts to activate. The final output is a weighted sum of the outputs from these active experts, where weights are derived by re-normalizing the scores ($ ext{softmax}(H W)$). This allows for massive parameter counts with low compute cost.
-
SFT Loss Function
3920s
The loss function minimizes the negative log likelihood of predicting $Y$ given $X$: $ ext{minimize } - rac{1}{N} rac{1}{I} ext{log } P(Y | X)$. This is applied over a sequence of tokens, allowing for supervised training on instruction-answer pairs.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.