Containment Is Lying to You: Measuring AI Agents Like Employees — Druid AI
The presentation critiques the reliance on single metrics, such as 'containment rate,' for evaluating AI agent performance. Druid AI introduces a comprehensive, multi-pillar framework to assess agent effectiveness. This framework scores agents across 26 components, categorized into six pillars (Configuration, Runtime, Governance, Mandate, Platform, and Outcomes). The scoring system uses three levels (0, 1, or 2) and weights components by business impact, emphasizing governance, risk, and compliance over simple conversation metrics.
Key takeaways
-
Containment Rate is Misleading
2:42
Relying solely on containment rate (the percentage of conversations handled without human intervention) can reward agents that fail to escalate complex, high-value issues (like mortgage refinancing) to senior human bankers. A holistic view is necessary.
-
The Seven-Pillar Framework
5:55
Agent evaluation requires a framework spanning six pillars: Configured (what the agent is built with), Runtime (what it executes), Governance (error tolerance, risk, compliance, observability, hallucination, tone, sentiment), Mandate (scope, business case, budget), Platform, and Outcomes. Governance is weighted highest (39%).
-
Scoring Components (0, 1, 2)
7:25
Each of the 26 components is scored on three levels: 0 (missing), 1 (measurable/observable, even manually), or 2 (managed/tracked and analyzed over time).
-
Measuring Business Success (ROI)
21:20
Effective evaluation moves beyond technical metrics to measure business outcomes. The framework helps quantify the ROI by determining what the agent takes off the human team's plate, allowing the business to focus on higher-value tasks.