Containment Is Lying to You: Measuring AI Agents Like Employees — Druid AI
Summary
The presentation critiques the reliance on single metrics, such as 'containment rate,' for evaluating AI agent performance. Druid AI introduces a comprehensive, multi-pillar framework to assess agent effectiveness. This framework scores agents across 26 components, categorized into six pillars (Configuration, Runtime, Governance, Mandate, Platform, and Outcomes). The scoring system uses three levels (0, 1, or 2) and weights components by business impact, emphasizing governance, risk, and compliance over simple conversation metrics.
Key takeaways
-
Containment Rate is Misleading
2:42
Relying solely on containment rate (the percentage of conversations handled without human intervention) can reward agents that fail to escalate complex, high-value issues (like mortgage refinancing) to senior human bankers. A holistic view is necessary.
-
The Seven-Pillar Framework
5:55
Agent evaluation requires a framework spanning six pillars: Configured (what the agent is built with), Runtime (what it executes), Governance (error tolerance, risk, compliance, observability, hallucination, tone, sentiment), Mandate (scope, business case, budget), Platform, and Outcomes. Governance is weighted highest (39%).
-
Scoring Components (0, 1, 2)
7:25
Each of the 26 components is scored on three levels: 0 (missing), 1 (measurable/observable, even manually), or 2 (managed/tracked and analyzed over time).
-
Measuring Business Success (ROI)
21:20
Effective evaluation moves beyond technical metrics to measure business outcomes. The framework helps quantify the ROI by determining what the agent takes off the human team's plate, allowing the business to focus on higher-value tasks.
Technical details
-
Agent Evaluation Framework
355s
The framework is built on six pillars: Configured, Runtime, Governance, Mandate, Platform, and Outcomes. Governance components include error tolerance, risk, compliance, observability, accuracy, hallucination, tone, and sentiment.
-
Scoring Methodology
445s
Components are scored 0 (missing), 1 (observable/measurable), or 2 (managed/tracked over time). The governance cluster is weighted at 39%, followed by Mandate (26%).
-
Agent Mandate Components
770s
The agent's mandate is defined by several critical boundaries: accepted tolerance, risk definition, compliance (e.g., SOC 2), isolated context (allowed documents/systems), and security (limited access to specific endpoints).
-
Runtime Logging and Validation
1140s
At runtime, agents must log tasks, input, and required context. Success criteria must be defined and linked to tasks to validate if the outcome (e.g., contract renewal) was successful.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.