# Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

## Executive summary

This workshop provides a deep dive into evaluating high-stakes AI agents, particularly for regulated industries like finance. The core message is that traditional observability is insufficient for non-deterministic LLM systems. The speakers outline the systematic 'AI Engineering Loop,' which mandates continuous improvement through structured practices: detailed tracing, dataset curation, running experiments, and implementing robust evaluation (Evals) that can be integrated into CI/CD pipelines to prevent regressions before deployment.

## Key takeaways

- The Problem with Non-Deterministic AI: AI agents can fail silently; system metrics may look 'green' (e.g., latency is fine), but the agent may fail to execute critical steps (like issuing a refund) or hallucinate actions. This necessitates specialized evaluation methods beyond traditional monitoring.
- The AI Engineering Loop: Continuous improvement requires a systematic loop: 1) Tracing (logging all steps, inputs, and outputs); 2) Monitoring (finding aggregate signals and failure modes); 3) Building Datasets (creating representative test cases); 4) Running Experiments (varying prompts, models, or tools); and 5) Evaluation (measuring quality).
- Implementing Quality Gates in CI/CD: The process can be automated by integrating evaluations into the CI/CD pipeline. By triggering a CI job on any prompt change, the system can run experiments against a defined dataset and fail the build if quality scores drop or guardrails are violated.
- Hybrid Evaluation Approach: For maximum reliability, a mix of evaluators is recommended: LLM-as-judge (using one LLM to score another's output), deterministic code evaluators (for exact matches), and custom guardrail checks (e.g., blocking investment advice).

## Technical details

- Tracing and Data Masking: Tracing captures the entire execution path, including root traces, individual steps, tool calls, and metadata. For regulated industries, PII identification and masking are critical, ensuring sensitive data (like credit card numbers) is redacted before persistence.
- Dataset Curation and Annotation: Datasets serve as the 'test cases' for the agent. Ground truth data can be built via expert curation, shadow runs (running the agent parallel to human processes), or by annotating production traces. Domain experts provide feedback by scoring outputs against defined failure modes (e.g., missing citations).
- Evaluation Methods: Evaluation approaches include LLM-as-judge (using an LLM to assess quality) and deterministic code evaluators (for 100% accurate checks). The goal is to establish measurable 'scores' for quality dimensions like source recall, language match, and compliance.
- System Architecture and Deployment: The platform supports self-hosted deployments and utilizes ClickHouse for data storage due to the massive scale of observed events. Data can be consumed via API, CLI, or MCP.

## Practical implications

- Implement automated quality gates in your CI/CD pipeline to prevent prompt or configuration changes from degrading agent performance.
- Prioritize comprehensive tracing and PII masking, especially when dealing with regulated data (finance, healthcare).
- Adopt a hybrid evaluation strategy, combining the flexibility of LLM-as-judge with the reliability of deterministic code checks.
- Use production traces and human expert annotation to continuously refine and expand your ground truth dataset.

## Topics

AI Agents, LLM Evaluation (Evals), Observability, CI/CD, Prompt Engineering, Regulated Industries, Langfuse, AIE CODE summit, Langfuse Academy

Source: https://www.youtube.com/watch?v=obfPzbapMIM