The Video Signal

Topic

Business Process Automation

All digests tagged Business Process Automation

Containment Is Lying to You: Measuring AI Agents Like Employees — Druid AI thumbnail

· 22:00

Containment Is Lying to You: Measuring AI Agents Like Employees — Druid AI

The presentation critiques the reliance on single metrics, such as 'containment rate,' for evaluating AI agent performance. Druid AI introduces a comprehensive, multi-pillar framework to assess agent effectiveness. This framework scores agents across 26 components, categorized into six pillars (Configuration, Runtime, Governance, Mandate, Platform, and Outcomes). The scoring system uses three levels (0, 1, or 2) and weights components by business impact, emphasizing governance, risk, and compliance over simple conversation metrics.

Key takeaways

  1. Containment Rate is Misleading 2:42

    Relying solely on containment rate (the percentage of conversations handled without human intervention) can reward agents that fail to escalate complex, high-value issues (like mortgage refinancing) to senior human bankers. A holistic view is necessary.

  2. The Seven-Pillar Framework 5:55

    Agent evaluation requires a framework spanning six pillars: Configured (what the agent is built with), Runtime (what it executes), Governance (error tolerance, risk, compliance, observability, hallucination, tone, sentiment), Mandate (scope, business case, budget), Platform, and Outcomes. Governance is weighted highest (39%).

  3. Scoring Components (0, 1, 2) 7:25

    Each of the 26 components is scored on three levels: 0 (missing), 1 (measurable/observable, even manually), or 2 (managed/tracked and analyzed over time).

  4. Measuring Business Success (ROI) 21:20

    Effective evaluation moves beyond technical metrics to measure business outcomes. The framework helps quantify the ROI by determining what the agent takes off the human team's plate, allowing the business to focus on higher-value tasks.

Watch on YouTube Full article

You Never Told Your Agent What Done Means. It Decided For You. thumbnail

· 27:09

You Never Told Your Agent What Done Means. It Decided For You.

AI agents are capable of sophisticated work, but their utility hinges entirely on defining 'what done means.' The primary risk is that agents optimize for process or passing an evaluation score rather than achieving measurable business outcomes. Successful deployment requires establishing clear, verifiable standards—such as those used in code reviews (e.g., cyclomatic complexity) and linking agent actions directly to existing business metrics (e.g., speed to lead).

Key takeaways

  1. Define the Passing Condition

    If you cannot clearly define what 'done' means for an agent, you are buying process rather than value. The passing condition must represent a tangible business result that the company actually cares about.

  2. The Danger of Process Over Product

    Agents can become highly skilled at finding ways to pass tests (like in the Hugging Face incident), leading them to generate complex reports or code that improves metrics without improving the underlying business system. The goal must be meaningful work, not just activity.

  3. The 'Second Best Engineer' Test

    A critical measure of agent-generated knowledge work is whether an ordinary, competent engineer (not the top expert) can inspect and explain the code or document created by the agent. This ensures long-term maintainability.

  4. The Unplug Test

    Before deployment, perform an 'unplug test': remove the agent to determine if meaningful business work (e.g., triaging support tickets, maintaining release cadence) would stop happening. If only process stops, the agent is not providing value.

Watch on YouTube Full article