7 Most Common AI Eval Mistakes (From Teaching 5K+ Engineers & PMs)
This guide outlines seven critical mistakes teams make when designing AI evaluations (Evals), emphasizing that effective Evals must be grounded in real-world data rather than generic assumptions. Key recommendations include shifting from generic, scored metrics (e.g., 1-5 Likert scales) to specific, binary (pass/fail) outcomes, ensuring domain experts perform data annotation, and rigorously validating LLM judges against human ground truth using proper data splitting techniques.
Key takeaways
-
Mistake #1: Creating Evals without data context
0:23
Do not brainstorm potential failures in isolation. Evals must be grounded in actual observed data to define 'what good looks like.' For example, an AI leasing assistant should be evaluated on specific failures, such as handling sales objections, rather than general conversational flow.
-
Mistake #2: Using generic, off-the-shelf metrics
1:56
Generic metrics (e.g., helpfulness, conciseness score) are non-actionable. Instead, define specific, observable metrics (e.g., 'Did we handle sales objection when a qualifying opportunity was presented?') and aim for binary pass/fail outcomes.
-
Mistake #3: Not validating LLM judges against human labels
4:29
To trust an LLM judge, segment your data into Train (20%), Dev (40%), and Test (40%). Tune the LLM judge on the Dev set, and measure its generalization capability on the held-out Test set. Focus on measuring Precision and Recall (TPR/TNR), not just accuracy.
-
Mistake #4: Not designing the product for Evals success
6:12
Design AI products, especially data agents, for the expert user persona. The interface should show intermediate calculations, source citations, and allow step-by-step review, making the system inherently more testable and auditable.
-
Mistake #5: Using Likert scales (1-5) with LLM judges
7:43
Avoid continuous scoring. Likert scales introduce massive complexity and require extensive human alignment. Focus instead on scoping Evals to a single, specific error and achieving a binary pass/fail outcome.
-
Mistake #6: Having the wrong people review the data
9:45
Data annotation and review must be performed by domain experts (e.g., lawyers for legal assistants, doctors for medical assistants), not the developers, who lack domain knowledge.
-
Mistake #7: Relying too much on automated evals
10:52
Automated agents (like coding agents) can miss context-specific errors (e.g., sales objections, markdown formatting, human handoff issues). Manual data review is essential to define the boundaries of 'good' performance.