Don't Trust Your AI Agents. Test Them Like a Scientist — Stephanie Jarmak, Omni Analytics
Summary
The presentation argues that most AI agent evaluations (evals) are prone to misinterpretation and should not be trusted blindly. Drawing on a background in astronomy and research science, the speaker emphasizes that evaluating agents requires treating the process like a scientific experiment: designing realistic tasks, running repeated tests, and focusing on comprehensive observability (full traces) rather than single metrics. The ultimate truth for agent performance remains production deployment.
Key takeaways
-
Be skeptical of evaluation results.
Evaluation metrics can be highly sensitive to changes in the evaluation harness or setup. For example, swapping only the harness in a benchmark can cause a 30%+ difference in reported effectiveness, meaning the configuration must reflect the actual deployment environment.
-
Public benchmarks are for shortlisting, not decision-making.
5:51
Public benchmarks provide a broad understanding of the landscape but should not be over-indexed upon. Evaluations must be tailored to the specific business needs and actual workloads of the organization.
-
Design evals like a scientific experiment.
8:54
To ensure reliable results, evaluations must involve realistic tasks, repeated runs (due to non-deterministic systems), and careful consideration of how different variables interact.
-
Focus on full observability and traces.
11:45
It is crucial to retain all traces and available context throughout the agent's pipeline to understand why a tool was not used or why an intended outcome failed, even if the final output seems correct.
-
Production is the final test.
While internal testing is vital, the only reliable measure of an agent's performance is its behavior in the actual production environment. Shadow deployment can help mitigate risk when making changes.
Technical details
-
Agent Evaluation Pitfalls
111s
The speaker demonstrated that changing the initial prompt given to an agent could yield a 28% difference in cost metrics, but this variance did not change the core failure (the agent still failed to use the intended tool). This highlights that superficial metrics can mislead.
-
Benchmark Sensitivity
247s
A comparison using Turbo Bench 4 showed that swapping only the evaluation harness resulted in a 30%+ difference in effectiveness between configurations (e.g., Opus 5 vs. Astra), emphasizing the variable impact of the testing setup.
-
Quality Gates and Safety
948s
When implementing automated quality gates, it is critical to decouple the system's ability to flag an issue from the mechanism that enforces a stop. Furthermore, organizations must assess the capacity for human review, as a low false alarm rate (e.g., 1%) can still generate an overwhelming number of required reviews (e.g., 10,000).
-
Trace Reliability
The agent's internal record (trace) can be untrustworthy. Researchers demonstrated that an independent record comparison was necessary to detect injected security issues that the agent's internal hash check failed to notice.
Mentioned resources
- Omni Analytics
- AIE CODE summit
- GitHub repo (Skills)
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.