# Don't Trust Your AI Agents. Test Them Like a Scientist — Stephanie Jarmak, Omni Analytics

## Executive summary

The presentation argues that most AI agent evaluations (evals) are prone to misinterpretation and should not be trusted blindly. Drawing on a background in astronomy and research science, the speaker emphasizes that evaluating agents requires treating the process like a scientific experiment: designing realistic tasks, running repeated tests, and focusing on comprehensive observability (full traces) rather than single metrics. The ultimate truth for agent performance remains production deployment.

## Key takeaways

- Be skeptical of evaluation results.: Evaluation metrics can be highly sensitive to changes in the evaluation harness or setup. For example, swapping only the harness in a benchmark can cause a 30%+ difference in reported effectiveness, meaning the configuration must reflect the actual deployment environment.
- Public benchmarks are for shortlisting, not decision-making.: Public benchmarks provide a broad understanding of the landscape but should not be over-indexed upon. Evaluations must be tailored to the specific business needs and actual workloads of the organization.
- Design evals like a scientific experiment.: To ensure reliable results, evaluations must involve realistic tasks, repeated runs (due to non-deterministic systems), and careful consideration of how different variables interact.
- Focus on full observability and traces.: It is crucial to retain all traces and available context throughout the agent's pipeline to understand why a tool was not used or why an intended outcome failed, even if the final output seems correct.
- Production is the final test.: While internal testing is vital, the only reliable measure of an agent's performance is its behavior in the actual production environment. Shadow deployment can help mitigate risk when making changes.

## Technical details

- Agent Evaluation Pitfalls: The speaker demonstrated that changing the initial prompt given to an agent could yield a 28% difference in cost metrics, but this variance did not change the core failure (the agent still failed to use the intended tool). This highlights that superficial metrics can mislead.
- Benchmark Sensitivity: A comparison using Turbo Bench 4 showed that swapping only the evaluation harness resulted in a 30%+ difference in effectiveness between configurations (e.g., Opus 5 vs. Astra), emphasizing the variable impact of the testing setup.
- Quality Gates and Safety: When implementing automated quality gates, it is critical to decouple the system's ability to flag an issue from the mechanism that enforces a stop. Furthermore, organizations must assess the capacity for human review, as a low false alarm rate (e.g., 1%) can still generate an overwhelming number of required reviews (e.g., 10,000).
- Trace Reliability: The agent's internal record (trace) can be untrustworthy. Researchers demonstrated that an independent record comparison was necessary to detect injected security issues that the agent's internal hash check failed to notice.

## Practical implications

- When designing automated testing pipelines for AI agents, treat the evaluation setup (harness, prompts, context) as a variable that must be rigorously controlled and documented.
- Implement multi-layered safety mechanisms (quality gates) that enforce stops, rather than merely flagging issues, and model the human review bottleneck capacity.
- Prioritize comprehensive observability by retaining full traces and context history, rather than relying solely on single, high-level performance metrics (like F1 score).
- Use production data and shadow deployment modes to validate agent performance, as this represents the most accurate measure of system reliability.

## Topics

AI Agent Evaluation, LLM Governance, Software Testing, Observability, Build Reliability, Omni Analytics, AIE CODE summit, GitHub repo (Skills)

Source: https://www.youtube.com/watch?v=A3FL_zTQKU0