Stanford CS229 Machine Learning | Spring 2026 | Lecture 16: Basic Concept in RL, Policy Gradient
This lecture provides an advanced deep dive into optimizing Transformer architectures for efficiency and adapting Large Language Models (LLMs) for various downstream tasks. Key focus areas include reducing the quadratic complexity of attention through techniques like Grouped Query Attention (GQA) and Sliding Window Attention; scaling models using Mixture of Experts (MoE) to decouple memory from compute; and exploring prompt-based methods such as In-Context Learning, Few-Shot, and Zero-Shot learning for task adaptation without updating model parameters.
Key takeaways
-
Efficiency in Attention Mechanisms
20:04
The standard self-attention mechanism has $O(T^2)$ complexity (where T is sequence length). To mitigate this, techniques like Grouped Query Attention (GQA) reduce the number of keys and values used across heads by mapping multiple query groups to a smaller set of shared keys/values. Similarly, Sliding Window Attention limits attention to only recent history, reducing complexity to $O(T imes W)$ where W is the window size.
-
Scaling with Mixture of Experts (MoE)
42:28
MoE allows models to have a large total parameter count (e.g., 30B) while keeping the active computation small (e.g., 3B). This is achieved by using a routing module that directs an input vector to only a subset of specialized expert sub-networks, significantly improving compute efficiency.
-
LLM Adaptation via Prompting
56:48
For downstream tasks (e.g., sentiment analysis), models can be adapted using In-Context Learning (ICL). This involves concatenating task examples and the test input into the prompt sequence without updating model parameters, which is fundamentally different from traditional fine-tuning.
-
Supervised Fine-Tuning (SFT)
1:04:00
SFT involves collecting data in an instruction/answer pair format and training the model by minimizing the negative log likelihood of predicting the answer ($Y$) given the instruction ($X$). This is a supervised process that updates the model's weights.