What a $900M Bank Error Teaches Us About AI Agents - John Feminella, Two Sigma

Key takeaways

1. The Problem of Failed Controls 1:43

   The 2020 Revlon incident demonstrated that even with multiple procedures, warning pop-ups, and approvers, human operators can override controls, leading to massive financial errors ($900M instead of $8M). This highlights that controls must account for non-deterministic human and agent behavior.

2. Agent Governance Must Be Composable

   Instead of monolithic controls, safety requires a mix of guardrails. The five proposed controls—Agent Registry, Risk-based Review, Tool Purpose, Shared Evals, and Classifiers—are designed to work together, allowing for high autonomy at an appropriate level of safety.

3. The Agent Registry (Control 1) 12:12

   To govern agents, they must first be visible. The registry makes agents legible by recording key facts: ownership, deployed tools, and endpoints. This prevents 'shadow IT' and allows the firm to use a single source of truth for agent capabilities.

4. Risk-Based Review (Control 2) 15:47

   Review processes must be tailored to the agent's risk profile. Applying the same manual review process to a low-risk summarizer and a high-risk production deployment agent creates bottlenecks and slows down valuable, small-scale experiments.

5. Tools Must Follow Purpose (Control 3)

   Agents should only be allowed to use tools and access endpoints strictly necessary for their declared purpose. This prevents agents from performing actions outside their intended scope, avoiding the need for overly restrictive, useless blanket blocks.

6. Shared Evaluation and Classification (Controls 4 & 5)

   Evaluation (Evals) must be standardized across the firm to maintain quality without creating review bottlenecks. Furthermore, instead of requiring human approval for every action (leading to 'review fatigue'), independent classifiers should handle routine, predictable calls, escalating only ambiguous or dangerous requests to humans.