The Video Signal

Topic

Stanford CS329A Self-Improving AI Agents

All digests tagged Stanford CS329A Self-Improving AI Agents

· 1:12:59

Stanford CS329A Self-Improving AI Agents | Part 3 | Robust Verification

This lecture traces the evolution of verification methods for Large Language Model (LLM) outputs, aiming to close the generation-verification gap. The discussion covers four major research advancements: training verifiers using outcome-based reward models (GSM8K), implementing process-based supervision via PRMs (PRM800K), automating annotation with Math-Shepherd, and finally, combining multiple weak verifiers into a robust system using Weaver. Key findings highlight that process supervision is generally superior to outcome supervision, and ensembling multiple specialized verifiers significantly boosts model accuracy on complex reasoning tasks.

Key takeaways

  1. Process Supervision vs. Outcome Supervision 26:00

    While outcome-based reward models (ORM) only check the final answer's correctness, process-based reward models (PRM) assign rewards per step of reasoning. PRMs are superior because they manage false positives better and encourage interpretable, human-endorsed steps [2:36:00].

  2. Ensembling Weak Verifiers (Weaver) 23:02

    The Weaver approach combines multiple weak verifiers (e.g., LLM judges, reward models) using techniques like Naive Bayes or logistic regression to create a single, highly capable verifier. This method significantly improves performance by leveraging the collective signal of diverse sources [3:42:00].

  3. Data Efficiency and Scaling 26:00

    PRMs are shown to be more data-efficient than ORMs. Furthermore, the lecture demonstrates that scaling verification by increasing the number of verifiers (rather than just sampling more completions) can improve results while maintaining computational efficiency [3:42:00].

  4. The Role of Self-Improvement 17:36

    Advanced techniques involve using the model itself to generate data (e.g., Math-Shepherd) and then training a PRM on this synthetic, semi-automated data, allowing the system to self-improve its reasoning capabilities [2:56:00].

Watch on YouTube Full article

· 1:12:39

Stanford CS329A Self-Improving AI Agents | Part 6 | Train Time Scaling/Scaling RL

This lecture explores advanced techniques for scaling Large Language Model (LLM) reasoning capabilities through 'train-time scaling' and Reinforcement Learning (RL). The discussion covers three key papers: STaR (Self-Taught Reasoner), DeepSeekMath, and DAPO. Key findings demonstrate that smaller models can achieve high accuracy on complex benchmarks like AIME by leveraging structured training data generation (STaR) or advanced RL algorithms (DAPO/GRPO). The core insight is that closing the feedback loop—using model outputs to improve the model itself—is crucial for boosting reasoning, especially in domains with verifiability.

Key takeaways

  1. Train-Time Scaling vs. Test-Time Scaling 2:00

    While test-time scaling (inference-based techniques like majority voting) improves accuracy by sampling outputs, train-time scaling uses the model's own filtered outputs to fine-tune and improve the model weights directly, creating a powerful closed feedback loop.

  2. STaR Boosts Reasoning via Rationalization 6:30

    The STaR method bootstraps reasoning by generating solutions on a small set of examples. It filters for correct answers and then generates rationales (hints) for incorrect attempts, allowing the model to learn from failed paths iteratively.

  3. GRPO Addresses RL Memory Constraints 10:05

    DeepSeekMath introduced Group Relative Policy Optimization (GRPO), an efficient alternative to PPO that reduces memory overhead by using a group baseline instead of maintaining multiple policy copies, enabling scaling RL to larger models.

  4. DAPO Stabilizes Complex Reasoning 17:30

    DAPO addresses training instability in long chain-of-thought reasoning by implementing asymmetric clipping (allowing bigger increases) and dynamic sampling (filtering out zero or one reward groups to maintain a useful gradient signal).

Watch on YouTube Full article