# Stanford CS229 Machine Learning | Spring 2026 | Lecture 16: Basic Concept in RL, Policy Gradient

## Executive summary

This lecture provides an advanced deep dive into optimizing Transformer architectures for efficiency and adapting Large Language Models (LLMs) for various downstream tasks. Key focus areas include reducing the quadratic complexity of attention through techniques like Grouped Query Attention (GQA) and Sliding Window Attention; scaling models using Mixture of Experts (MoE) to decouple memory from compute; and exploring prompt-based methods such as In-Context Learning, Few-Shot, and Zero-Shot learning for task adaptation without updating model parameters.

## Key takeaways

- Efficiency in Attention Mechanisms: The standard self-attention mechanism has $O(T^2)$ complexity (where T is sequence length). To mitigate this, techniques like Grouped Query Attention (GQA) reduce the number of keys and values used across heads by mapping multiple query groups to a smaller set of shared keys/values. Similarly, Sliding Window Attention limits attention to only recent history, reducing complexity to $O(T imes W)$ where W is the window size.
- Scaling with Mixture of Experts (MoE): MoE allows models to have a large total parameter count (e.g., 30B) while keeping the active computation small (e.g., 3B). This is achieved by using a routing module that directs an input vector to only a subset of specialized expert sub-networks, significantly improving compute efficiency.
- LLM Adaptation via Prompting: For downstream tasks (e.g., sentiment analysis), models can be adapted using In-Context Learning (ICL). This involves concatenating task examples and the test input into the prompt sequence without updating model parameters, which is fundamentally different from traditional fine-tuning.
- Supervised Fine-Tuning (SFT): SFT involves collecting data in an instruction/answer pair format and training the model by minimizing the negative log likelihood of predicting the answer ($Y$) given the instruction ($X$). This is a supervised process that updates the model's weights.

## Technical details

- Transformer Attention (QKV): Attention relies on Query (Q), Key (K), and Value (V) vectors. The core calculation involves $Q K^T / ext{scale} + ext{mask}$ followed by Softmax, then multiplication by V. For multiple heads, the process is replicated with different weights.
- KV Cache and Memory Bottleneck: During generation (inference), only K and V need to be stored (the KV cache). This memory requirement limits the maximum batch size, often causing the GPU to become memory-bound rather than compute-bound.
- Grouped Query Attention (GQA): GQA reduces redundancy by having multiple query heads ($N_h$) map to a smaller number of shared key and value groups ($N_g$), where $N_g = N_h / ext{tall}$. This significantly reduces the memory footprint required for the KV cache.
- Mixture of Experts (MoE) Routing: The routing module determines a subset $S$ of experts to activate. The final output is a weighted sum of the outputs from these active experts, where weights are derived by re-normalizing the scores ($ ext{softmax}(H W)$). This allows for massive parameter counts with low compute cost.
- SFT Loss Function: The loss function minimizes the negative log likelihood of predicting $Y$ given $X$: $ ext{minimize } - rac{1}{N} rac{1}{I} ext{log } P(Y | X)$. This is applied over a sequence of tokens, allowing for supervised training on instruction-answer pairs.

## Practical implications

- Architects can design highly efficient models by implementing GQA or MoE to manage memory and compute constraints, enabling larger parameter counts without prohibitive inference latency.
- LLMs can be quickly adapted for new tasks (e.g., classification) using prompt engineering (ICL/Few-Shot) rather than requiring expensive full fine-tuning cycles.
- The shift from traditional model deployment to prompting allows companies to use a single 'off-the-shelf' LLM and customize it via prompts, simplifying the MLOps pipeline.

## Topics

Transformer Architecture, Attention Mechanisms (Self-Attention), Model Optimization (GQA, MoE), LLM Prompt Engineering, Supervised Fine-Tuning (SFT), Stanford CS229 Machine Learning, Stanford AI Programs Info

Source: https://www.youtube.com/watch?v=hHC-SF3utxg