# What a $900M Bank Error Teaches Us About AI Agents - John Feminella, Two Sigma

## Executive summary

Drawing lessons from a $900 million banking error (the Revlon payment mistake), this talk outlines five composable controls necessary to grant AI agents real autonomy in highly regulated enterprise environments. The core challenge is balancing innovation (allowing agents to operate) with safety (preventing catastrophic failures). The proposed solution moves beyond simple sandboxing or manual approvals by implementing systemic guardrails, including an Agent Registry, risk-based review, purpose-constrained tool access, shared evaluation methodologies, and advanced classifiers to minimize human review fatigue.

## Key takeaways

- The Problem of Failed Controls: The 2020 Revlon incident demonstrated that even with multiple procedures, warning pop-ups, and approvers, human operators can override controls, leading to massive financial errors ($900M instead of $8M). This highlights that controls must account for non-deterministic human and agent behavior.
- Agent Governance Must Be Composable: Instead of monolithic controls, safety requires a mix of guardrails. The five proposed controls—Agent Registry, Risk-based Review, Tool Purpose, Shared Evals, and Classifiers—are designed to work together, allowing for high autonomy at an appropriate level of safety.
- The Agent Registry (Control 1): To govern agents, they must first be visible. The registry makes agents legible by recording key facts: ownership, deployed tools, and endpoints. This prevents 'shadow IT' and allows the firm to use a single source of truth for agent capabilities.
- Risk-Based Review (Control 2): Review processes must be tailored to the agent's risk profile. Applying the same manual review process to a low-risk summarizer and a high-risk production deployment agent creates bottlenecks and slows down valuable, small-scale experiments.
- Tools Must Follow Purpose (Control 3): Agents should only be allowed to use tools and access endpoints strictly necessary for their declared purpose. This prevents agents from performing actions outside their intended scope, avoiding the need for overly restrictive, useless blanket blocks.
- Shared Evaluation and Classification (Controls 4 & 5): Evaluation (Evals) must be standardized across the firm to maintain quality without creating review bottlenecks. Furthermore, instead of requiring human approval for every action (leading to 'review fatigue'), independent classifiers should handle routine, predictable calls, escalating only ambiguous or dangerous requests to humans.

## Technical details

- Financial Transaction Processing: The Revlon loan transaction utilized internal processing software called Flexcube. The error occurred when human operators overrode warning notifications, causing the entire outstanding principal ($900M) to be sent to 300 lenders instead of the intended interest payment ($8M).
- Agent Architecture and Governance: The proposed architecture relies on five composable controls: 1) Agent Registry (for visibility); 2) Risk-based Review (for appropriate scrutiny); 3) Purpose-constrained Tool Access (for limiting blast radius); 4) Shared Evals (for consistent quality assurance); and 5) Classifiers (for reducing human review burden).
- Agent Capabilities and Constraints: Agents are non-deterministic systems. To manage risk, they must declare their purpose upfront. The system enforces boundaries by translating policies (from the registry) into enforcible statements regarding allowed endpoints and resources.

## Practical implications

- For build engineers, this mandates designing governance layers (like an Agent Registry) that are integrated into the CI/CD pipeline for AI agents, treating agent code and dependencies with the same rigor as traditional software.
- The concept of 'composable controls' suggests that governance should be modular, allowing teams to apply only the necessary safety checks (e.g., low-risk agents skip heavy manual review).
- Implementing purpose-constrained tool access requires robust runtime policy enforcement, ensuring that an agent's declared purpose dictates its allowed API calls and resource access.

## Topics

AI Governance, Agent Architecture, System Reliability, Risk Management, Software Engineering Controls, John Feminella, Two Sigma, AIE CODE summit

Source: https://www.youtube.com/watch?v=iniryju23Sk