# From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI

## Executive summary

The talk outlines a shift in AI agent evaluation from simple production 'traces' to controlled, repeatable 'simulations.' To reliably develop and deploy agents, companies must build private benchmarks that mimic their specific production environment, tools, and policies. These simulations allow engineers to test the entire agent stack—including cost, latency, and policy adherence—and integrate this process into a formal CI pipeline for continuous improvement (Agent Ops).

## Key takeaways

- Private Benchmarks are Essential for Production Readiness: Public benchmarks (e.g., WebArena) are useful for general orientation but fail to capture domain-specific use cases, internal tooling, or company policies. A private benchmark is necessary because it allows comparison on critical metrics like cost per task and latency, not just success rate.
- Simulation Enables Full Stack Evaluation: Offline simulation turns production traces into repeatable experiments. This allows testing the full agent stack (model, prompt, skills, tools) while keeping the environment and evaluators constant between runs, enabling apples-to-apples comparisons.
- Benchmark Development Requires CI/CD Discipline: The benchmark itself must be treated as software. It requires a dedicated CI pipeline to ensure dependencies are pinned, fixtures are present, and the core Oracle solution passes before it can be used for agent testing.

## Technical details

- Benchmark Anatomy: A benchmark task consists of several components: `instruction.md` (the prompt), the `environment` (e.g., Dockerfile/Docker Compose defining APIs, databases, and tools), the `Oracle solution` (a ground truth run to ensure solvability), verifiers, and metadata.
- Simulation Mechanics: The simulation process involves an agent interacting with a controlled environment. Verifiers analyze not just the final output, but also the full state change, including database states, API responses, and artifacts generated during the run.
- Evaluation Methods: Verification can use deterministic checks (for simple outputs) or advanced methods like using an LLM as a judge to evaluate the quality of the agent's planning and trace adherence. Subject Matter Experts (SMEs) are reserved for resolving discrepancies between verifiers.

## Practical implications

- Engineers should establish a dedicated 'Agent Ops' loop where observability traces (failures) are used to expand the private benchmark, which then serves as the input for simulation experiments. This ensures fixes are applied at the correct layer (e.g., updating skills or tools) rather than just modifying the prompt.
- The process of building and maintaining benchmarks must be formalized using software engineering practices, including dependency management and dedicated CI pipelines.

## Topics

Agent Ops, AI Agents, Simulation Benchmarking, CI/CD Pipeline, LLM Judge, Reproducibility, Snorkel AI, WebArena, SweepBench

Source: https://www.youtube.com/watch?v=Ib5t2RLtxvM