# 1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life

## Executive summary

The presentation analyzes 1,368 AI incidents to propose a structured, risk-based framework for evaluating AI systems. The core argument is that AI failure must be assessed holistically across three dimensions—Technical Strength, Deployment Controls, and Business Context—rather than focusing solely on model performance. The analysis emphasizes that weak deployment controls (e.g., human review, monitoring) are the most critical factor tied to harm. The proposed methodology involves decomposing risk into four components and adopting a 'build-break-fix' loop, which mandates that evaluations must attempt to falsify system claims rather than confirm them.

## Key takeaways

- Three Dimensions of AI Failure: AI systems must be evaluated across three core dimensions: 1) Technical Strength (model issues, data quality, fairness); 2) Deployment Controls (authority, human mediation, exposure control, monitoring); and 3) Business Context (financial impact, legal consequences, scale of users affected).
- Prioritizing Failure Sub-Dimensions: Analysis of 1,368 incidents showed that while 13 sub-dimensions cover 98% of failures, deployment controls are found to be more critically associated with harm than technical failure alone. Evaluation efforts should prioritize sub-dimensions based on their potential for harm.
- Failure-Driven Evaluation (Build-Break-Fix): Evaluations must adopt a failure-driven approach, inspired by the need to avoid confirmation bias. This involves starting with a system claim, actively challenging it to find failures, and iteratively implementing supporting systems or controls until a desired level of confidence is achieved.

## Technical details

- Risk Decomposition: Risk should be decomposed into four components: how often the model fails, whether that failure reaches a real decision, whether it goes uncaught, and how severe the consequences are. This dictates that the evaluation must cover the entire deployment setup, not just the model.
- Agentic System Testing: Testing agentic systems requires expanding beyond traditional ML metrics (accuracy, AUC ROC) to test the entire deployment system. Specific tests must validate tool responses, action sequences (trajectories), human review effectiveness, and the system's ability to stay within defined permissions or boundaries.
- Materiality Scale: A materiality scale allows organizations to prioritize AI use cases based on the potential harm associated with their failure, enabling the allocation of resources to the highest-risk systems first.

## Practical implications

- Build engineers should shift from frequency-based to risk-based evaluation strategies.
- System design must incorporate explicit controls (e.g., human-in-the-loop, monitoring) to contain failures before they reach production.
- Implement a 'build-break-fix' cycle during testing to actively attempt to falsify system claims and map out high-risk regions.

## Topics

AI System Evaluation, Risk Management, Failure Analysis, System Architecture, AI Governance, AI incident database, AIE CODE summit

Source: https://www.youtube.com/watch?v=mMw3arwQsL0