How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
Summary
This talk addresses the challenge of evaluating highly specific AI agents, particularly in domains like contract review or specialized Q&A, where public benchmarks are unavailable and domain experts are too scarce for manual grading. The core solution presented is to reverse the evaluation process: instead of asking the agent a question and grading the answer, the process starts with a known correct answer and generates the necessary input (the test case) for the agent to find it. This methodology drastically accelerates testing, moving from slow, expert-dependent weekly reviews to dozens of automated daily tests, and allows for the creation of robust, scalable test sets.
Key takeaways
-
The Answer-First Evaluation Paradigm
When traditional benchmarking fails due to lack of public data or expert time, reverse the problem. Start with the desired output (the answer) and generate the input (the question/context) that forces the agent to reproduce it. This allows for automated, code-based grading.
-
Scaling from Manual to Automated Testing
10:33
Applying this 'answer-first' approach transformed a RAG project from requiring one expert-graded test per week to running dozens of automated tests daily, significantly improving the speed and reliability of the evaluation cycle.
-
Five Methods for Generating Test Sets (Contract Review)
17:19
For complex tasks like supplier contract review, five specific techniques were used to build comprehensive test sets: 1) Past decisions (comparing old vs. new contracts); 2) Planted rule violations (generating contracts that intentionally break known rules); 3) Removed clauses (testing for missing information); 4) Changes that shouldn't matter (testing robustness to minor changes like supplier name); and 5) Templates with blanks (generating variations using structured templates).
Technical details
-
RAG Evaluation Improvement
141s
In a previous telecom RAG project, the team found that while complex pipelines were added, the real bottleneck was the manual expert grading. By reversing the problem (generating a question from a known correct document/answer), they could automate the grading, leading to increased accuracy and speed.
-
Agent Architecture (Air Canada)
1008s
The contract agent involves four steps: 1) Spotting the clause in the contract; 2) Finding relevant documents to support an amendment; 3) Writing the amendment; and 4) Classification (deciding if the amendment should be kept).
-
Code-Based Grading vs. LLM Judges
It is strongly recommended to use deterministic code checks (e.g., exact match, code check) whenever possible, as relying on LLM judges introduces significant noise and complexity, making the evaluation process unstable and difficult to optimize.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.