The Agent Works. Does Anyone Trust It? — Michele Grant, Leading at Scale
Summary
While an AI agent's technical accuracy (e.g., matching analyst reports) proves functionality, it does not guarantee organizational trust or measurable business value. Michele Grant outlines a framework for moving agents from pilot to core workflow by answering three critical questions: What value does it create? Who is accountable for it? And what evidence permits greater autonomy? Successful deployment requires establishing pre-deployment baselines, implementing robust governance (like the Three Lines Model), and maintaining rigorous technical artifacts to ensure the agent operates within defined risk and human oversight boundaries.
Key takeaways
-
Value Requires a Baseline
7:57
Before deployment, measurable value must be established using pre-deployment baselines for three metrics: Cycle Time (time to compile reports), Analyst Capacity (hours saved, net of compute/licensing costs), and Detection Time (time from data slip to committee visibility).
-
Accountability via Three Lines Model
10:29
Accountability must be applied to the agent using the Three Lines Model: Management owns the controls (the integration lead); Risk and Compliance oversee the agent's exposure; and Internal Audit provides assurance based on material risk.
-
Governance through Artifacts
To grant autonomy, three artifacts are required: an Auditable Decision Log (recording raw data, tool calls, and human sign-off); a Model Inventory Record (documenting data lineage, dependencies, and regression triggers); and an Autonomy Scorecard (tracking progress through stages like Shadow Mode and Exception Ensemble).
Technical details
-
Deterministic Code Requirements
Core mathematical calculations and reconciliation rules must run in deterministic, finance-approved code, rather than being generated as conversational prose from an LLM. The model's written summary should be treated as a helpful explanation, not proof.
-
Agent Autonomy Stages
Autonomy advances through four stages: Stage 1 (Shadow Mode - agent runs alongside manual process); Stage 2 (Supervised Draft - senior analyst reviews every line item); Stage 3 (Exception Ensemble - analyst reviews lag initiatives plus random sample); and Stage 4 (Exceptions Only - analyst reviews only high-risk anomalies).
-
Risk Monitoring
When operating in advanced stages, monitoring error severity and tracking the override rate is critical. A falling override rate may indicate a slip in the depth of review rather than improved performance.
Mentioned resources
- Leading at Scale
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.