From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
This workshop details the methodology for building robust, reproducible, and comprehensive evaluation pipelines for AI agents, moving beyond the limitations of single average scores. The speaker introduces WolfBench, a framework that treats performance as a distribution, requiring the tracking of metrics like consistency, best/worst runs, and total cost. The process emphasizes controlling all variables—from the benchmark dataset and agent type to sandbox resources and inference settings—to ensure that reported scores accurately reflect dependable behavior rather than lucky, single-run successes.
Key takeaways
-
Performance is a Distribution, Not a Point
6:52
A single average score is insufficient for evaluating AI agents. Reliable evaluation requires analyzing the distribution of results, including the best, worst, and average runs, to assess variance and consistency.
-
The Importance of Uniform Conditions
12:04
To ensure comparable results, all variables must be kept uniform. Changing any variable—such as the benchmark version, agent, or sandbox resource limits—constitutes a new benchmark and invalidates direct comparisons.
-
The Six Layers of Evaluation
15:53
A comprehensive evaluation pipeline must manage six swappable layers: benchmark (dataset), runner (harness), sandbox, agent, model service, and settings. Controlling all these layers is crucial for reproducibility.
-
Analyzing Reliability vs. Capability
27:10
The leaderboard should be interpreted by looking at the 'solid base' (percentage solved in every run) for consistency, the 'average' (the usual reported score), and the 'ceiling' (tasks solved at least once) for maximum capability.