From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI
Summary
The talk outlines a shift in AI agent evaluation from simple production 'traces' to controlled, repeatable 'simulations.' To reliably develop and deploy agents, companies must build private benchmarks that mimic their specific production environment, tools, and policies. These simulations allow engineers to test the entire agent stack—including cost, latency, and policy adherence—and integrate this process into a formal CI pipeline for continuous improvement (Agent Ops).
Key takeaways
-
Private Benchmarks are Essential for Production Readiness
6:41
Public benchmarks (e.g., WebArena) are useful for general orientation but fail to capture domain-specific use cases, internal tooling, or company policies. A private benchmark is necessary because it allows comparison on critical metrics like cost per task and latency, not just success rate.
-
Simulation Enables Full Stack Evaluation
2:00
Offline simulation turns production traces into repeatable experiments. This allows testing the full agent stack (model, prompt, skills, tools) while keeping the environment and evaluators constant between runs, enabling apples-to-apples comparisons.
-
Benchmark Development Requires CI/CD Discipline
The benchmark itself must be treated as software. It requires a dedicated CI pipeline to ensure dependencies are pinned, fixtures are present, and the core Oracle solution passes before it can be used for agent testing.
Technical details
-
Benchmark Anatomy
634s
A benchmark task consists of several components: `instruction.md` (the prompt), the `environment` (e.g., Dockerfile/Docker Compose defining APIs, databases, and tools), the `Oracle solution` (a ground truth run to ensure solvability), verifiers, and metadata.
-
Simulation Mechanics
920s
The simulation process involves an agent interacting with a controlled environment. Verifiers analyze not just the final output, but also the full state change, including database states, API responses, and artifacts generated during the run.
-
Evaluation Methods
1054s
Verification can use deterministic checks (for simple outputs) or advanced methods like using an LLM as a judge to evaluate the quality of the agent's planning and trace adherence. Subject Matter Experts (SMEs) are reserved for resolving discrepancies between verifiers.
Mentioned resources
- Snorkel AI
- WebArena
- SweepBench
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.