# Stanford CS329A Self-Improving AI Agents | Part 3 | Robust Verification

## Executive summary

This lecture traces the evolution of verification methods for Large Language Model (LLM) outputs, aiming to close the generation-verification gap. The discussion covers four major research advancements: training verifiers using outcome-based reward models (GSM8K), implementing process-based supervision via PRMs (PRM800K), automating annotation with Math-Shepherd, and finally, combining multiple weak verifiers into a robust system using Weaver. Key findings highlight that process supervision is generally superior to outcome supervision, and ensembling multiple specialized verifiers significantly boosts model accuracy on complex reasoning tasks.

## Key takeaways

- Process Supervision vs. Outcome Supervision: While outcome-based reward models (ORM) only check the final answer's correctness, process-based reward models (PRM) assign rewards per step of reasoning. PRMs are superior because they manage false positives better and encourage interpretable, human-endorsed steps [2:36:00].
- Ensembling Weak Verifiers (Weaver): The Weaver approach combines multiple weak verifiers (e.g., LLM judges, reward models) using techniques like Naive Bayes or logistic regression to create a single, highly capable verifier. This method significantly improves performance by leveraging the collective signal of diverse sources [3:42:00].
- Data Efficiency and Scaling: PRMs are shown to be more data-efficient than ORMs. Furthermore, the lecture demonstrates that scaling verification by increasing the number of verifiers (rather than just sampling more completions) can improve results while maintaining computational efficiency [3:42:00].
- The Role of Self-Improvement: Advanced techniques involve using the model itself to generate data (e.g., Math-Shepherd) and then training a PRM on this synthetic, semi-automated data, allowing the system to self-improve its reasoning capabilities [2:56:00].

## Technical details

- GSM8K (OpenAI, 2021): Introduced the GSM8K dataset for multi-step math reasoning. The initial verifier trained on this dataset used a binary loss (correct/incorrect) and a language modeling objective to predict solution correctness at the token level [0:45:00].
- Outcome-based vs. Process-based Reward Models: ORM assigns reward based on the final solution's correctness (outcome). PRM assigns a separate reward per step of the reasoning process, requiring human annotation of stepwise labels (PRM800K dataset) [1:35:00].
- Math-Shepherd (Automated Annotation): Addresses the need for human annotations by defining step quality based on potential—either a 'hard estimate' (if *any* N subsequent generations reach the correct answer) or a 'soft estimate' (frequency of reaching the correct answer) [2:56:00].
- Weaver Ensemble Architecture: A method to reduce the generation-verification gap by combining multiple weak verifiers (e.g., LLM judges, reward models). It uses weak supervision and techniques like Naive Bayes or logistic regression to assign weighted scores, significantly outperforming naive ensembling [3:20:00].

## Practical implications

- Implementing verification layers (Verifiers) as a quality gate for LLM outputs, especially in complex reasoning tasks.
- Using process supervision (PRMs) to ensure that the *reasoning steps* are correct, not just the final answer, which is critical for debugging and reliability in build systems.
- Adopting ensemble methods (like Weaver) to increase robustness by combining multiple specialized verifiers rather than relying on a single model's judgment.
- Recognizing that verification can be used as a reward signal for Reinforcement Learning (RL fine-tuning), allowing the generator model to self-improve its reasoning process.

## Topics

Large Language Models (LLMs), Verification, Reward Modeling, Reinforcement Learning (RL), Ensemble Methods, Test-Time Scaling, Reasoning Benchmarks, GSM8K Dataset, PRM800K Dataset, Stanford CS329A Self-Improving AI Agents

Source: https://www.youtube.com/watch?v=p7TdPUcPoik