# Can Claude Build Your Evals?

## Executive summary

The video provides a critical review of Anthropic's Claude skill for automating AI evaluation (eval) design and hill climbing. While the tool is highly promising for its ability to guide users through complex eval principles, the speakers argue that the automated workflow is overly prescriptive, suffers from poor user experience (e.g., requiring manual labeling in markdown files), and forces a suboptimal process (eval design before deep data introspection). The consensus is that a human expert would gain more by simply building a data viewer and manually reviewing the raw traces than by following the automated eval design process.

## Key takeaways

- Principles of Good Eval Design: Good evals should not saturate the model (i.e., performance should be well below 100%) and should focus on 'passable headroom at the frontier.' Adversarial sampling is recommended to target model failure points rather than just cases humans find difficult.
- Automated Workflow Critique: The `build eval` command guides users through sampling inputs (production transcripts, bug reports, etc.) and proposing a grader (programmatic verification or LLM as a judge). However, the process is criticized for its high friction, particularly the requirement to review and label data within a markdown file, which is poor UX.
- Data-First Approach is Superior: The speakers strongly advocate for a 'data-first' approach. They argue that a human expert would be better served by immediately building a data viewer and manually introspecting the raw traces, rather than being guided into building an eval before fully understanding the data's failure modes.

## Technical details

- Eval Design Principles: The skill highlights principles such as ensuring evals mirror production, and that the most capable model at the highest effort should score well below 100% to allow for reliable performance measurement.
- Claude API Skill Usage: The process involves using the `build eval` command with a `traces.json` file (e.g., 100 traces from an apartment leasing assistant). The skill analyzes the data and identifies failure modes, such as 'call transfer issues' or 'tour scheduling' errors.
- Grader and Runner Output: The system proposes a grader (e.g., checking for the string 'Would you like me to connect you?') and runs a baseline, providing a score with a confidence interval. The output is structured as JSONL and per-case transcripts.
- Workflow Flaws: The process often bundles multiple, distinct issues (e.g., tone, transfer protocol, and confirmation questions) into a single, uninterpretable eval, making debugging difficult.

## Practical implications

- When designing AI evaluations, prioritize manual data introspection (building a data viewer) over relying solely on automated eval design tools.
- Be critical of automated tools that bundle multiple, distinct failure modes into a single evaluation, as this makes the resulting eval uninterpretable.
- For build engineers, the process of creating a data viewer or annotation app might be more valuable than the automated eval design process itself.

## Topics

AI Evaluation (Evals), LLM Grading, Build Engineering, Data Annotation, Prompt Engineering, AI Evals October 2026 cohort, Evals Guide

Source: https://www.youtube.com/watch?v=5ykulgyB8kk