# Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 4 - Reinforcement Learning with LLMs

## Executive summary

This lecture provides a deep technical dive into Reinforcement Learning (RL) methods for post-training Large Language Models (LLMs), focusing on Preference Tuning. The core objective is to maximize expected rewards by training a policy ($ ext{LLM}$) based on human preferences (RLHF) or verifiable outcomes (RLVR). Key advanced techniques discussed include Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), which address issues like high gradient variance and policy deviation. Newer methods, such as On-Policy Distillation (OPD), propose using distribution comparison (e.g., reverse KL divergence) to achieve token-level supervision without needing a separate value function.

## Key takeaways

- RL Objective and Gradient Calculation: The goal is to maximize the expected reward, which requires computing the gradient of the objective function. This gradient is mathematically derived using the identity involving the gradient of the log of the policy ($ abla ext{log} ext{pi}_ heta$), incentivizing the model to increase probabilities for high-reward actions and decrease them for low-reward actions. (00:12:23)
- PPO and GRPO: Addressing Variance and Stability: PPO (Proximal Policy Optimization) maximizes advantages and includes a proximity term (P) to prevent the policy from deviating too much from the old policy ($ ext{pi}_ ext{old}$). GRPO (Group Relative Policy Optimization) achieves similar goals by computing advantages relative to the mean and standard deviation of multiple generated responses, eliminating the need for a separate value function. (01:07:29)
- On-Policy Distillation (OPD): OPD proposes using the reverse KL divergence to supervise the student model's next token prediction based on a Teacher model's distribution. This provides dense, token-level supervision without relying on a reward model, making it highly efficient under compute constraints. (01:39:32)

## Technical details

- RLHF vs. RLVR: Reward signals fall into two categories: 1) Preference Tuning (RLHF), where rewards are subjective and quantified using human preferences (e.g., Bradley-Terry formulation). 2) Verifiable Rewards (RLVR), where the answer can be deterministically checked (e.g., math or coding problems). (00:06:35)
- Policy Constraints and Divergence: To prevent the trained policy ($ ext{pi}_ heta$) from deviating too much from the initial base policy ($ ext{pi}_ ext{ref}$) or the policy used for data generation ($ ext{pi}_ ext{old}$), methods use constraints like clipping the importance sampling ratio ($ ho$) or adding a KL divergence penalty ($ ext{KL}( ext{pi}_ heta || ext{pi}_ ext{ref})$). (00:53:03)
- Value Function and Advantage: The value function is a model trained jointly with the policy to estimate the expected future rewards starting from a partial completion. The advantage function is calculated by comparing the expected value up to a token versus the expected value without that token, providing token-level credit assignment. (00:42:49)
- Reverse KL Divergence: In OPD, the reverse KL divergence is used to measure the difference between the Teacher model's distribution and the Student model's distribution, penalizing the student when its probability weights deviate significantly from the teacher's. (02:00:00)

## Practical implications

- RL-based methods are crucial for aligning LLMs with human preferences and improving reasoning capabilities post-SFT.
- OPD offers a compute-efficient alternative to traditional RL methods by using distillation, which is particularly useful when training on resource-constrained hardware.
- The use of multiple teachers (Multi-Teacher OPD) is recommended to prevent forgetting and ensure alignment across multiple objectives.

## Topics

Reinforcement Learning (RL), Large Language Models (LLMs), Policy Optimization, Preference Tuning (RLHF), Model Distillation, Optimization Theory, Stanford CME295 Syllabus

Source: https://www.youtube.com/watch?v=3Vaj0Oka_t4