From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI
The talk outlines a shift in AI agent evaluation from simple production 'traces' to controlled, repeatable 'simulations.' To reliably develop and deploy agents, companies must build private benchmarks that mimic their specific production environment, tools, and policies. These simulations allow engineers to test the entire agent stack—including cost, latency, and policy adherence—and integrate this process into a formal CI pipeline for continuous improvement (Agent Ops).
Key takeaways
-
Private Benchmarks are Essential for Production Readiness
6:41
Public benchmarks (e.g., WebArena) are useful for general orientation but fail to capture domain-specific use cases, internal tooling, or company policies. A private benchmark is necessary because it allows comparison on critical metrics like cost per task and latency, not just success rate.
-
Simulation Enables Full Stack Evaluation
2:00
Offline simulation turns production traces into repeatable experiments. This allows testing the full agent stack (model, prompt, skills, tools) while keeping the environment and evaluators constant between runs, enabling apples-to-apples comparisons.
-
Benchmark Development Requires CI/CD Discipline
The benchmark itself must be treated as software. It requires a dedicated CI pipeline to ensure dependencies are pinned, fixtures are present, and the core Oracle solution passes before it can be used for agent testing.