# From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases

## Executive summary

This workshop details the methodology for building robust, reproducible, and comprehensive evaluation pipelines for AI agents, moving beyond the limitations of single average scores. The speaker introduces WolfBench, a framework that treats performance as a distribution, requiring the tracking of metrics like consistency, best/worst runs, and total cost. The process emphasizes controlling all variables—from the benchmark dataset and agent type to sandbox resources and inference settings—to ensure that reported scores accurately reflect dependable behavior rather than lucky, single-run successes.

## Key takeaways

- Performance is a Distribution, Not a Point: A single average score is insufficient for evaluating AI agents. Reliable evaluation requires analyzing the distribution of results, including the best, worst, and average runs, to assess variance and consistency.
- The Importance of Uniform Conditions: To ensure comparable results, all variables must be kept uniform. Changing any variable—such as the benchmark version, agent, or sandbox resource limits—constitutes a new benchmark and invalidates direct comparisons.
- The Six Layers of Evaluation: A comprehensive evaluation pipeline must manage six swappable layers: benchmark (dataset), runner (harness), sandbox, agent, model service, and settings. Controlling all these layers is crucial for reproducibility.
- Analyzing Reliability vs. Capability: The leaderboard should be interpreted by looking at the 'solid base' (percentage solved in every run) for consistency, the 'average' (the usual reported score), and the 'ceiling' (tasks solved at least once) for maximum capability.

## Technical details

- Evaluation Pipeline Architecture: The pipeline is composed of six layers: benchmark, runner, sandbox, agent, model service, and settings. The process requires using a dedicated evaluation harness (e.g., Harbor) to run tasks in isolated sandboxes, preventing system interference.
- Benchmark Selection: The speaker recommends using agentic benchmarks like `Terminal Bench` (currently using version 2.0) for testing agent capabilities, as it involves tasks like fixing Git repositories or repairing databases. For reproducibility, sticking to a specific, stable version is critical.
- Infrastructure and Tools: The recommended tools include `Harbor` as the evaluation harness, running in a dedicated environment (e.g., `UV` tool install), and utilizing cloud providers' serverless sandboxes for isolation. The system requires an API key and a YAML configuration file.
- Critical Configuration Variables: Many variables affect results and must be explicitly controlled: Context window size (e.g., 256K for GLM 5.2), Thinking effort (e.g., 'extra high' vs. 'medium'), System prompt (must define the agent's role), Tool access, Sandbox resources (CPU/memory), Task timeout, Concurrency, and Retry policy.
- Cost and Efficiency Analysis: Evaluation must consider cost and efficiency. A cheaper, faster model might consume more tokens, leading to a higher total cost than a larger, more expensive model. The analysis must track tokens used and total cost alongside the score.

## Practical implications

- Adopt a 'build-engineering' mindset for AI evaluation: treat the entire evaluation setup (model, agent, settings, sandbox) as a complex, version-controlled configuration.
- Implement rigorous testing protocols that track not only the final score but also resource consumption (tokens, cost) and run consistency (variance).
- When comparing models, always document the full configuration (including model versions, provider routing, and system prompts) to ensure reproducibility and transparency.
- Use dedicated tools like WolfBench to aggregate and visualize results, allowing stakeholders to filter and compare metrics like consistency and maximum capability.

## Topics

AI Agents, Model Evaluation, Reproducibility, Build Engineering, CI/CD, LLM Benchmarking, Weights & Biases (W&B), WolfBench, Harbor, Terminal Bench

Source: https://www.youtube.com/watch?v=YIxzNZDDy1A