Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 4 - Reinforcement Learning with LLMs

Summary

This lecture provides a deep technical dive into Reinforcement Learning (RL) methods for post-training Large Language Models (LLMs), focusing on Preference Tuning. The core objective is to maximize expected rewards by training a policy ($ ext{LLM}$) based on human preferences (RLHF) or verifiable outcomes (RLVR). Key advanced techniques discussed include Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), which address issues like high gradient variance and policy deviation. Newer methods, such as On-Policy Distillation (OPD), propose using distribution comparison (e.g., reverse KL divergence) to achieve token-level supervision without needing a separate value function.

Technical details

• RLHF vs. RLVR 635s

  Reward signals fall into two categories: 1) Preference Tuning (RLHF), where rewards are subjective and quantified using human preferences (e.g., Bradley-Terry formulation). 2) Verifiable Rewards (RLVR), where the answer can be deterministically checked (e.g., math or coding problems). (00:06:35)

• Policy Constraints and Divergence 5303s

  To prevent the trained policy ($ ext{pi}_ heta$) from deviating too much from the initial base policy ($ ext{pi}_ ext{ref}$) or the policy used for data generation ($ ext{pi}_ ext{old}$), methods use constraints like clipping the importance sampling ratio ($ ho$) or adding a KL divergence penalty ($ ext{KL}( ext{pi}_ heta || ext{pi}_ ext{ref})$). (00:53:03)

• Value Function and Advantage 4249s

  The value function is a model trained jointly with the policy to estimate the expected future rewards starting from a partial completion. The advantage function is calculated by comparing the expected value up to a token versus the expected value without that token, providing token-level credit assignment. (00:42:49)

• Reverse KL Divergence 1200s

  In OPD, the reverse KL divergence is used to measure the difference between the Teacher model's distribution and the Student model's distribution, penalizing the student when its probability weights deviate significantly from the teacher's. (02:00:00)

Mentioned resources

• Stanford CME295 Syllabus (Course Material)

Channel & topics

• Stanford Online
• Reinforcement Learning (RL)
• Large Language Models (LLMs)
• Policy Optimization
• Preference Tuning (RLHF)
• Model Distillation
• Optimization Theory
• Stanford CME295 Syllabus