1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
Summary
The presentation analyzes 1,368 AI incidents to propose a structured, risk-based framework for evaluating AI systems. The core argument is that AI failure must be assessed holistically across three dimensions—Technical Strength, Deployment Controls, and Business Context—rather than focusing solely on model performance. The analysis emphasizes that weak deployment controls (e.g., human review, monitoring) are the most critical factor tied to harm. The proposed methodology involves decomposing risk into four components and adopting a 'build-break-fix' loop, which mandates that evaluations must attempt to falsify system claims rather than confirm them.
Key takeaways
-
Three Dimensions of AI Failure
AI systems must be evaluated across three core dimensions: 1) Technical Strength (model issues, data quality, fairness); 2) Deployment Controls (authority, human mediation, exposure control, monitoring); and 3) Business Context (financial impact, legal consequences, scale of users affected).
-
Prioritizing Failure Sub-Dimensions
Analysis of 1,368 incidents showed that while 13 sub-dimensions cover 98% of failures, deployment controls are found to be more critically associated with harm than technical failure alone. Evaluation efforts should prioritize sub-dimensions based on their potential for harm.
-
Failure-Driven Evaluation (Build-Break-Fix)
Evaluations must adopt a failure-driven approach, inspired by the need to avoid confirmation bias. This involves starting with a system claim, actively challenging it to find failures, and iteratively implementing supporting systems or controls until a desired level of confidence is achieved.
Technical details
-
Risk Decomposition
0s
Risk should be decomposed into four components: how often the model fails, whether that failure reaches a real decision, whether it goes uncaught, and how severe the consequences are. This dictates that the evaluation must cover the entire deployment setup, not just the model.
-
Agentic System Testing
0s
Testing agentic systems requires expanding beyond traditional ML metrics (accuracy, AUC ROC) to test the entire deployment system. Specific tests must validate tool responses, action sequences (trajectories), human review effectiveness, and the system's ability to stay within defined permissions or boundaries.
-
Materiality Scale
0s
A materiality scale allows organizations to prioritize AI use cases based on the potential harm associated with their failure, enabling the allocation of resources to the highest-risk systems first.
Mentioned resources
- AI incident database
- AIE CODE summit
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.