# How We Built an Agent That Improves Itself — Zubin Aysola, Weights & Biases

## Executive summary

The presentation details the architecture of ARIA, a self-improving AI agent developed by Weights & Biases. The core innovation is a robust, closed-loop evaluation framework that allows the agent to perform 'auto-research' by taking production traces, converting them into offline evaluation tasks, identifying bugs (e.g., missing SDK calls), writing fixes, and benchmarking the improved variant against the production version. This system ensures that both production and research environments are byte-for-byte identical, creating a continuous 'eval flywheel' for agent improvement.

## Key takeaways

- The Covariance Challenge in Agent Evaluation: A major challenge is that benchmarks, evaluations, and agent configurations are all covariant. To maintain airtight measurement, the team keeps the production and research agents byte-for-byte identical (5:50).
- The Eval Flywheel: The system creates a continuous improvement loop: every production miss or win is captured as a new task, which is then used to train and improve the agent, minimizing manual benchmark creation (12:03).
- Offline Evaluation Framework: The framework uses a dedicated offline evaluation sandbox to simulate environments and run agent variants. This process involves converting production traces into structured evaluation tasks (3:40).
- Dual Scoring Mechanism: ARIA scores itself in two ways: pass/fail (normative) and relative comparison, allowing comparison between different agent variants (10:14).

## Technical details

- Agent Architecture & Framework: The system uses a model-agnostic harness defined by YAML variants to run multiple parallel agent configurations. This allows for testing multiple mutations of the exact same configuration (7:15).
- Production/Research Sync: The research and production codebases are kept identical, utilizing a 4-hour sync job to prevent drift when new skills or variants are developed (5:50).
- Trace Processing: Production traces logged in Weights & Biases Weave are ripped into the offline evaluation framework, enabling the agent to 'hill climb' on real-world errors (3:40).
- Sandbox Environment: The system requires an unconstrained sandbox environment to allow the agent to perform parallel executions of its own research loop and discover emergent behaviors (8:04).

## Practical implications

- For teams building complex AI agents, adopting a unified platform (like W&B) for both production tracing and offline evaluation is critical for maintaining measurement integrity.
- The concept of the 'eval flywheel' suggests that operational failures (production misses) should be systematically captured and converted into structured training tasks, accelerating the development cycle.
- Maintaining byte-for-byte parity between production and research environments is essential for reliable agent benchmarking.

## Topics

AI Agents, Evaluation Frameworks, MLOps, Self-Improvement, Weights & Biases, Weights & Biases (W&B), W&B Weave, ARIA Agent

Source: https://www.youtube.com/watch?v=XyV6bSMyq-I