From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Summary
This workshop details the methodology for building robust, reproducible, and comprehensive evaluation pipelines for AI agents, moving beyond the limitations of single average scores. The speaker introduces WolfBench, a framework that treats performance as a distribution, requiring the tracking of metrics like consistency, best/worst runs, and total cost. The process emphasizes controlling all variables—from the benchmark dataset and agent type to sandbox resources and inference settings—to ensure that reported scores accurately reflect dependable behavior rather than lucky, single-run successes.
Key takeaways
-
Performance is a Distribution, Not a Point
6:52
A single average score is insufficient for evaluating AI agents. Reliable evaluation requires analyzing the distribution of results, including the best, worst, and average runs, to assess variance and consistency.
-
The Importance of Uniform Conditions
12:04
To ensure comparable results, all variables must be kept uniform. Changing any variable—such as the benchmark version, agent, or sandbox resource limits—constitutes a new benchmark and invalidates direct comparisons.
-
The Six Layers of Evaluation
15:53
A comprehensive evaluation pipeline must manage six swappable layers: benchmark (dataset), runner (harness), sandbox, agent, model service, and settings. Controlling all these layers is crucial for reproducibility.
-
Analyzing Reliability vs. Capability
27:10
The leaderboard should be interpreted by looking at the 'solid base' (percentage solved in every run) for consistency, the 'average' (the usual reported score), and the 'ceiling' (tasks solved at least once) for maximum capability.
Technical details
-
Evaluation Pipeline Architecture
953s
The pipeline is composed of six layers: benchmark, runner, sandbox, agent, model service, and settings. The process requires using a dedicated evaluation harness (e.g., Harbor) to run tasks in isolated sandboxes, preventing system interference.
-
Benchmark Selection
830s
The speaker recommends using agentic benchmarks like `Terminal Bench` (currently using version 2.0) for testing agent capabilities, as it involves tasks like fixing Git repositories or repairing databases. For reproducibility, sticking to a specific, stable version is critical.
-
Infrastructure and Tools
850s
The recommended tools include `Harbor` as the evaluation harness, running in a dedicated environment (e.g., `UV` tool install), and utilizing cloud providers' serverless sandboxes for isolation. The system requires an API key and a YAML configuration file.
-
Critical Configuration Variables
1130s
Many variables affect results and must be explicitly controlled: Context window size (e.g., 256K for GLM 5.2), Thinking effort (e.g., 'extra high' vs. 'medium'), System prompt (must define the agent's role), Tool access, Sandbox resources (CPU/memory), Task timeout, Concurrency, and Retry policy.
-
Cost and Efficiency Analysis
1750s
Evaluation must consider cost and efficiency. A cheaper, faster model might consume more tokens, leading to a higher total cost than a larger, more expensive model. The analysis must track tokens used and total cost alongside the score.
Mentioned resources
- Weights & Biases (W&B)
- WolfBench
- Harbor
- Terminal Bench
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.