# How To Decide Who Owns AI Evals

## Executive summary

AI evaluation (Evals) should not be treated as a separate discipline or dedicated role. Instead, it must be fundamentally integrated into the inner loop of the AI application development process. The core responsibility for driving evals belongs to the domain expert, not necessarily the developer. To manage complexity and ensure consistent quality, the speaker recommends establishing a 'benevolent dictator'—a trusted domain expert who has the final say on defining what constitutes 'good' performance. Success in evals relies heavily on creating custom annotation tools, often built using platforms like Codex or Cloud Code, to streamline data reading and analysis.

## Key takeaways

- Evals are an Inner Loop Process: Evals should not be a separate, siloed discipline. They are fundamentally part of the continuous process of building and improving an AI application. Treating them as a separate role risks inefficiency.
- Domain Expertise is Paramount: The person leading the evals must possess the deepest domain expertise. Developers should only lead evals if the application is specifically a coding application; otherwise, they are not the domain expert.
- Establish a 'Benevolent Dictator': To minimize complexity and conflicting labeling, establish a single, trusted domain expert (the 'benevolent dictator') who has the authority to make the final determination on what constitutes successful output.

## Technical details

- Annotation Tooling: To make the 'benevolent dictator' successful, custom annotation tools are required. These tools should be built (e.g., using Codex or Cloud Code) to render data and consolidate all necessary information for the annotator in one place.
- Error Analysis: A key task in evals is detailed error analysis, which requires deep domain knowledge to accurately identify failure modes in LLM products.

## Practical implications

- When designing ML Ops pipelines, ensure that the evaluation phase is tightly coupled with the development loop, rather than being treated as a separate QA gate.
- Prioritize identifying and empowering a single, highly knowledgeable domain expert to own the evaluation criteria and final judgment.
- Invest in building custom tooling (using internal code generation capabilities) to simplify the data annotation and error analysis process for the domain expert.

## Topics

AI Evaluation (Evals), ML Ops, Domain Expertise, Software Architecture, LLM Product Development, AI Evals October 2026 cohort, Free Evals Guide

Source: https://www.youtube.com/watch?v=V07jAHSsQOU