# Stanford CS329A Self-Improving AI Agents | Part 2 | Test-Time Compute Scaling

## Executive summary

The lecture details advanced methods for improving Large Language Model (LLM) performance through 'inference scaling' or 'test-time compute scaling,' rather than relying solely on expensive pre-training. Key techniques include repeated sampling (Large Language Monkeys), which shows that coverage follows a predictable power law with the number of samples. The discussion highlights the critical need for robust verification mechanisms to bridge the generation-verification gap, and concludes by introducing the Arkon framework, an architecture search method that optimizes complex inference pipelines using components like Fusion, Critic, and Ranker.

## Key takeaways

- Inference Scaling Paradigm Shift: LLM capability can be significantly enhanced at inference time by increasing compute (e.g., repeated sampling) without modifying model parameters or requiring fine-tuning, offering a new paradigm compared to traditional pre-training and fine-tuning.
- Repeated Sampling Effectiveness: By repeatedly querying the same problem (e.g., using Llama 3-8b or DeepSeek), selecting the correct response among candidates, models can achieve performance comparable to larger proprietary models like GPT-4o.
- The Role of Verification: For repeated sampling to be effective, automated verification is crucial. The 'generation-verification gap' describes the large difference between the best possible outcome (Oracle selection) and what can be achieved using simple methods like majority voting.
- Advanced Scaling Architectures: The Arkon framework treats inference scaling as an architecture design problem, optimizing the combination of techniques (e.g., Fusion, Critic, Ranker) to maximize accuracy given a limited compute budget.

## Technical details

- Test-Time Compute Scaling Laws: The relationship between problem coverage (solved by at least one sample) and the number of parallel samples ($k$) follows an exponential power law. This allows engineers to predict the required compute resources needed to achieve a target level of performance.
- Reward Models for Selection: Advanced selection methods include Outcome Reward Models (ORM), which score the final answer, and Process Reward Models (PRM), which assign a score to each step of the generation process. These guide search techniques like beam search.
- Arkon Inference Architecture Search: This framework optimizes inference by mixing and matching multiple components: Generation (sampling), Fusion (synthesizing one answer from $K$ responses), Critic (identifying weaknesses/strengths of a response), and Ranker (ranking generated answers). The optimization uses Bayesian methods to maximize accuracy given an inference budget.
- Unit Test Generation & Evaluation: For coding tasks, models can generate unit tests and then evaluate a generated solution against those tests. This provides a verifiable method for measuring correctness in complex domains.

## Practical implications

- Engineers can design complex LLM pipelines that dynamically allocate compute resources (tokens/samples) to maximize accuracy for specific tasks and difficulty levels.
- The ability to use open-source models, when combined with advanced test-time scaling techniques, allows them to match or exceed the performance of frontier closed-source models (e.g., GPT-4.0, Claude 3.5 Sonnet) on reasoning and coding benchmarks.
- Implementing a robust verification layer is critical; relying only on simple methods like majority voting will significantly underestimate true model capability.

## Topics

Large Language Models (LLMs), AI Agents, Scaling Laws, Inference Optimization, Machine Learning Architecture Search, Large Language Monkeys paper, SWE-bench, KernelBench

Source: https://www.youtube.com/watch?v=-Ggc37xLj_Y