# How to Evaluate Coding Agents

## Executive summary

Building rigorous evaluation systems for AI coding agents is highly complex. The talk emphasizes that relying on public benchmarks, single test runs, or simple cost comparisons is insufficient. Effective evaluation requires a deep focus on task selection (balancing representativeness vs. measurability), building highly observable harnesses, and understanding the multiplicative entropy introduced by components like the prompt, model, and reasoning level. The core message is that continuous iteration, deep trace analysis, and robust observability are necessary to achieve reliable measurements.

## Key takeaways

- Public Benchmarks are Insufficient: Existing public benchmarks (like SweeBench) are often not representative of an organization's specific codebase, frameworks (e.g., Django vs. Go), or internal definition of 'done.' Furthermore, since these benchmarks have been public for years, models are already trained on them.
- The Spectrum of Tasks: Representative vs. Measurable: Tasks exist on a spectrum: highly measurable tasks (e.g., passing a specific unit test) are easy to grade, but not necessarily representative of real-world development. Representative tasks introduce non-determinism and uncertainty, making them much harder to measure.
- Single Runs are Not Measurements: Measurements must account for variability. Aggregating results is crucial, as the delta between the best and worst agent can be misleading. Analyzing traces and understanding the root cause of failures (e.g., throttling vs. logic error) is more valuable than the final score.
- Observability is Paramount: The evaluation harness must be highly observable. Metrics should track not only correctness but also cost, latency, and the number of turns. Failure to account for grading time (which can dominate total latency) can severely dilute performance results.

## Technical details

- Task Selection & Prompting: To improve task quality, one must decide whether to prioritize measurability (e.g., fixing a failing unit test) or representativeness (mimicking real-world, ambiguous coding sessions). Prompts should mimic natural human conversation rather than being overly structured, as agents perform better in less constrained, more natural environments.
- Sandboxing and Isolation: To prevent agents from cheating or accessing external information, the environment must be contained (e.g., using Docker with minimal network access). However, achieving perfect isolation is impossible, especially when agents require network-facing tool calls (like web searches or Confluence access), which introduces significant entropy.
- Evaluation Harness Design: The harness must be designed to capture multiple dimensions: speed, correctness, cost, and policy adherence. It must emit detailed metrics and traces to ensure debuggability and accurate evaluation, rather than relying solely on the final output.

## Practical implications

- When designing CI/CD evaluation pipelines for AI agents, treat the evaluation harness itself as a critical, complex component that requires deep observability.
- Do not assume that simple, passing unit tests are sufficient for measuring agent capability; consider the ambiguity and context of real-world tasks.
- When reporting performance metrics, always aggregate results and analyze traces to understand variability, rather than relying on single-run 'flashy' numbers.
- Be aware that the complexity of the evaluation setup (prompt, model, harness, provider) is multiplicative; changing any single component can drastically alter the end-to-end result.

## Topics

Coding Agents, AI Evaluation, Build Observability, LLM Judging, CI/CD, Prompt Engineering, AI Evals Guide

Source: https://www.youtube.com/watch?v=1yAQxMACq0k