# How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI

## Executive summary

This workshop provides a hands-on guide for building robust AI evaluations (evals), moving beyond subjective 'vibe checks' to concrete, measurable quality gates. The process involves analyzing agent traces to define specific pass/fail criteria, utilizing specialized tools like the Prompt Playground, and implementing various evaluation types (e.g., LLM as a judge, Decision Models) to ensure agent performance can be objectively measured and integrated into the development lifecycle.

## Key takeaways

- The Necessity of Evals: Evals are crucial for detecting regressions, objectively comparing model versions, and establishing quality gates in CI, as relying on subjective 'vibe checks' does not scale. Without them, it is impossible to prove if an agent fix worked or if a failure has occurred. (02:06)
- Evals vs. Unit Tests: Unlike traditional unit tests (which have one input and one correct output), AI agents produce complex, multi-step outputs captured in 'traces.' Evals must analyze these traces to account for multiple valid outputs and complex logic. (03:08)
- Defining Success (The Criteria): Criteria must be highly specific and concrete (e.g., 'answers the follow-up directly using the trip's details') rather than vague (e.g., 'helpful' or 'good response'). It is recommended to focus on one primary criterion per eval, especially when using LLM as a judge. (19:03)
- The Evaluation Loop: Evals are iterative. The process involves: 1) Analyzing traces to identify spans to judge, 2) Defining concrete criteria, 3) Building the eval template in the Prompt Playground, and 4) Running the eval against a dataset of traces. (26:53)

## Technical details

- Traces and Observability: A trace is the comprehensive log of all individual steps (spans) an AI agent takes to generate a response. These spans include root spans, individual agent spans, and tool spans, providing visibility into the agent's internal decision-making process. (04:13, 14:20)
- Types of Evals: Key types include: LLM as a judge (using a secondary LLM to grade aspects like QA or summarization, providing labels and explanations); Decision Models (fast, cheap APIs for true/false or label probability); Code evals (for deterministic checks, e.g., checking JSON structure); Annotations (using golden datasets); and Agent as a judge (using one agent to evaluate another). (05:15)
- Building the Eval Template: The criteria are translated into a prompt template in the Prompt Playground. Variable mapping is used to connect the template to the specific attributes within the traces, ensuring flexibility regardless of how the data is named. (18:30)
- Advanced Evaluation Stages: Post-development, advanced stages include 'Signal' (managed agents finding issues and making fixes), 'Alignment' (annotating examples and comparing human feedback to eval results), and 'Online evals' (running evals continuously on every incoming trace for real-time monitoring). (32:08)

## Practical implications

- Always start by defining the success criteria for your agent using concrete, measurable statements, rather than vague goals.
- When integrating evals into CI/CD, focus on capturing the full agent trace (all spans) to ensure comprehensive testing of complex, multi-step logic.
- Use the Arize platform's tracing features to filter and select specific spans (e.g., root spans, tool calls) that are relevant to the failure mode you wish to test.
- Treat eval creation as an iterative process; start small (e.g., 20 traces, one dimension) and expand the number of evals as the application matures.

## Topics

AI Evaluation, LLM Agents, Observability, Prompt Engineering, CI/CD Testing, Decision Modeling, Arize AI Platform, Arize Prompt Playground, Notion One-Pager

Source: https://www.youtube.com/watch?v=gv4w23xYj-0