# How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada

## Executive summary

This talk addresses the challenge of evaluating highly specific AI agents, particularly in domains like contract review or specialized Q&A, where public benchmarks are unavailable and domain experts are too scarce for manual grading. The core solution presented is to reverse the evaluation process: instead of asking the agent a question and grading the answer, the process starts with a known correct answer and generates the necessary input (the test case) for the agent to find it. This methodology drastically accelerates testing, moving from slow, expert-dependent weekly reviews to dozens of automated daily tests, and allows for the creation of robust, scalable test sets.

## Key takeaways

- The Answer-First Evaluation Paradigm: When traditional benchmarking fails due to lack of public data or expert time, reverse the problem. Start with the desired output (the answer) and generate the input (the question/context) that forces the agent to reproduce it. This allows for automated, code-based grading.
- Scaling from Manual to Automated Testing: Applying this 'answer-first' approach transformed a RAG project from requiring one expert-graded test per week to running dozens of automated tests daily, significantly improving the speed and reliability of the evaluation cycle.
- Five Methods for Generating Test Sets (Contract Review): For complex tasks like supplier contract review, five specific techniques were used to build comprehensive test sets: 1) Past decisions (comparing old vs. new contracts); 2) Planted rule violations (generating contracts that intentionally break known rules); 3) Removed clauses (testing for missing information); 4) Changes that shouldn't matter (testing robustness to minor changes like supplier name); and 5) Templates with blanks (generating variations using structured templates).

## Technical details

- RAG Evaluation Improvement: In a previous telecom RAG project, the team found that while complex pipelines were added, the real bottleneck was the manual expert grading. By reversing the problem (generating a question from a known correct document/answer), they could automate the grading, leading to increased accuracy and speed.
- Agent Architecture (Air Canada): The contract agent involves four steps: 1) Spotting the clause in the contract; 2) Finding relevant documents to support an amendment; 3) Writing the amendment; and 4) Classification (deciding if the amendment should be kept).
- Code-Based Grading vs. LLM Judges: It is strongly recommended to use deterministic code checks (e.g., exact match, code check) whenever possible, as relying on LLM judges introduces significant noise and complexity, making the evaluation process unstable and difficult to optimize.

## Practical implications

- When building AI agents for niche, high-stakes domains, prioritize building automated, code-based test sets over relying on human expert review.
- Adopt the 'answer-first' methodology to generate test cases, ensuring that the evaluation process scales with the complexity of the task, not the availability of human time.
- For robust testing, design test sets that cover not only positive cases but also negative scenarios (e.g., planted rule violations, removed clauses, and non-impactful changes).

## Topics

AI Agents, Retrieval-Augmented Generation (RAG), Test Generation, Build Automation, Domain Expertise, Nathanael Weill, Air Canada, AIE CODE summit

Source: https://www.youtube.com/watch?v=c_SPW__jF9w