How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI
Summary
This workshop provides a hands-on guide for building robust AI evaluations (evals), moving beyond subjective 'vibe checks' to concrete, measurable quality gates. The process involves analyzing agent traces to define specific pass/fail criteria, utilizing specialized tools like the Prompt Playground, and implementing various evaluation types (e.g., LLM as a judge, Decision Models) to ensure agent performance can be objectively measured and integrated into the development lifecycle.
Key takeaways
-
The Necessity of Evals
2:06
Evals are crucial for detecting regressions, objectively comparing model versions, and establishing quality gates in CI, as relying on subjective 'vibe checks' does not scale. Without them, it is impossible to prove if an agent fix worked or if a failure has occurred. (02:06)
-
Evals vs. Unit Tests
3:18
Unlike traditional unit tests (which have one input and one correct output), AI agents produce complex, multi-step outputs captured in 'traces.' Evals must analyze these traces to account for multiple valid outputs and complex logic. (03:08)
-
Defining Success (The Criteria)
18:23
Criteria must be highly specific and concrete (e.g., 'answers the follow-up directly using the trip's details') rather than vague (e.g., 'helpful' or 'good response'). It is recommended to focus on one primary criterion per eval, especially when using LLM as a judge. (19:03)
-
The Evaluation Loop
26:53
Evals are iterative. The process involves: 1) Analyzing traces to identify spans to judge, 2) Defining concrete criteria, 3) Building the eval template in the Prompt Playground, and 4) Running the eval against a dataset of traces. (26:53)
Technical details
-
Traces and Observability
1420s
A trace is the comprehensive log of all individual steps (spans) an AI agent takes to generate a response. These spans include root spans, individual agent spans, and tool spans, providing visibility into the agent's internal decision-making process. (04:13, 14:20)
-
Types of Evals
315s
Key types include: LLM as a judge (using a secondary LLM to grade aspects like QA or summarization, providing labels and explanations); Decision Models (fast, cheap APIs for true/false or label probability); Code evals (for deterministic checks, e.g., checking JSON structure); Annotations (using golden datasets); and Agent as a judge (using one agent to evaluate another). (05:15)
-
Building the Eval Template
1530s
The criteria are translated into a prompt template in the Prompt Playground. Variable mapping is used to connect the template to the specific attributes within the traces, ensuring flexibility regardless of how the data is named. (18:30)
-
Advanced Evaluation Stages
Post-development, advanced stages include 'Signal' (managed agents finding issues and making fixes), 'Alignment' (annotating examples and comparing human feedback to eval results), and 'Online evals' (running evals continuously on every incoming trace for real-time monitoring). (32:08)
Mentioned resources
- Arize AI Platform
- Arize Prompt Playground
- Notion One-Pager
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.