# Containment Is Lying to You: Measuring AI Agents Like Employees — Druid AI

## Executive summary

The presentation critiques the reliance on single metrics, such as 'containment rate,' for evaluating AI agent performance. Druid AI introduces a comprehensive, multi-pillar framework to assess agent effectiveness. This framework scores agents across 26 components, categorized into six pillars (Configuration, Runtime, Governance, Mandate, Platform, and Outcomes). The scoring system uses three levels (0, 1, or 2) and weights components by business impact, emphasizing governance, risk, and compliance over simple conversation metrics.

## Key takeaways

- Containment Rate is Misleading: Relying solely on containment rate (the percentage of conversations handled without human intervention) can reward agents that fail to escalate complex, high-value issues (like mortgage refinancing) to senior human bankers. A holistic view is necessary.
- The Seven-Pillar Framework: Agent evaluation requires a framework spanning six pillars: Configured (what the agent is built with), Runtime (what it executes), Governance (error tolerance, risk, compliance, observability, hallucination, tone, sentiment), Mandate (scope, business case, budget), Platform, and Outcomes. Governance is weighted highest (39%).
- Scoring Components (0, 1, 2): Each of the 26 components is scored on three levels: 0 (missing), 1 (measurable/observable, even manually), or 2 (managed/tracked and analyzed over time).
- Measuring Business Success (ROI): Effective evaluation moves beyond technical metrics to measure business outcomes. The framework helps quantify the ROI by determining what the agent takes off the human team's plate, allowing the business to focus on higher-value tasks.

## Technical details

- Agent Evaluation Framework: The framework is built on six pillars: Configured, Runtime, Governance, Mandate, Platform, and Outcomes. Governance components include error tolerance, risk, compliance, observability, accuracy, hallucination, tone, and sentiment.
- Scoring Methodology: Components are scored 0 (missing), 1 (observable/measurable), or 2 (managed/tracked over time). The governance cluster is weighted at 39%, followed by Mandate (26%).
- Agent Mandate Components: The agent's mandate is defined by several critical boundaries: accepted tolerance, risk definition, compliance (e.g., SOC 2), isolated context (allowed documents/systems), and security (limited access to specific endpoints).
- Runtime Logging and Validation: At runtime, agents must log tasks, input, and required context. Success criteria must be defined and linked to tasks to validate if the outcome (e.g., contract renewal) was successful.

## Practical implications

- Do not use containment rate as the sole metric for AI agent success; it ignores business context and escalation value.
- Implement a comprehensive scorecard that tracks governance, risk, and compliance alongside technical performance.
- Define clear boundaries (Mandate) for agents, including acceptable error tolerance and limited system access, to mitigate risk.
- Measure agent success by quantifying the business value (ROI) and time saved for human employees, rather than just task completion rates.

## Topics

AI Agent Governance, AI Performance Metrics, Risk Management, Business Process Automation, AI Adoption Strategy, Druid AI, Druid 2026 AI Adoption Benchmark

Source: https://www.youtube.com/watch?v=PwCSYy-oGsM