# How To Get Your Team To Invest In Evals

## Executive summary

The most effective way to gain team buy-in and secure investment in AI evaluation (Evals) processes is not through academic arguments or hard selling, but by consistently demonstrating value through structured error analysis. By analyzing user traces, identifying top failure modes, quantifying their impact, and regularly presenting both the issues and the subsequent fixes, engineers can lead by example and build internal consensus around the necessity of robust evaluation.

## Key takeaways

- Prioritize Error Analysis over Advocacy: Instead of trying to convince the team of the theoretical value of evals, start by performing structured error analysis on 50 to 100 traces. This practical approach allows you to find concrete issues in the AI product.
- Quantify and Narrate Failure Modes: After identifying failure modes, quantify how often high-impact errors occur. Crucially, build a narrative around how users are interacting with the product in surprising ways, as this insight is often found only by reading raw traces.
- Demonstrate Value Through Fixes: Maintain a regular cadence of data analysis presentations: present the issues found, explain the narrative impact on business progress, and then show the specific steps taken to fix the error rate. This demonstrates a nuanced understanding of the product's limitations.

## Technical details

- Error Analysis Methodology: Error analysis is defined as a structured way of looking at data to find issues in an AI product. The process involves listing top failure modes, quantifying their frequency, and presenting the findings alongside the corrective actions.
- AI Product Evaluation (Evals): The value of evals is speculative, similar to classic machine learning projects. Convincing stakeholders requires proving value over time through continuous experimentation and data analysis, rather than relying on upfront ROI calculations.
- Code Benchmarking: Historically, benchmarks like CodeSearchNet were built as a set of evals for code understanding, which later served as groundwork for evaluating early coding models (e.g., OpenAI's first code embeddings).

## Practical implications

- When presenting findings, do not rely solely on dashboards and metrics; explain the 'why'—the user narrative and the business impact of the errors.
- When fixing an issue, always show the resulting reduction in the error rate to prove the efficacy of the intervention.
- To start, focus on deep, manual data analysis (reading traces) to generate initial, high-impact insights that drive the conversation.

## Topics

AI Product Development, Evaluation Metrics (Evals), Data Analysis, Machine Learning Operations (MLOps), Build Process Improvement, Free Evals Guide

Source: https://www.youtube.com/watch?v=5TNzt6wuD-s