Give the Agent a Budget, Not a Token — Sachin Malhotra, Anthropic
Summary
The presentation argues that granting AI agents 'unbounded power' via simple tokens is dangerous. Instead of narrowing the token scope (a boolean fix), engineers must implement a comprehensive 'budget' system with four dimensions: how much, how fast, what can be undone, and who notices. Key solutions include using asymmetric verbs (prioritizing loud failures), enforcing rate limits on writes, implementing trip wires over static allow lists, and utilizing an 'undo test' to size the overall safety policy.
Key takeaways
-
Budget vs. Token
7:03
A token is a boolean (yes/no) scope; a budget is multi-dimensional, considering volume, velocity, reversibility, and observability. The failure was giving the agent unbounded power, not the model itself.
-
Asymmetric Verbs
10:05
Prioritize granting agents access to operations that fail loudly (e.g., unskipping a test, which causes CI to go red) and keep critical failure verbs (like skipping a test) reserved for human intervention with an audit trail.
-
Rate Limits & Trip Wires
13:54
Implement rate limits on every write operation, ensuring the ceiling refills automatically. Use trip wires (monitoring aggregate behavior) instead of static allow lists, as trip wires adapt to real-world data.
-
The Undo Test
This test asks if the agent can autonomously roll back its own changes and what the blast radius would be if it failed. If not, a second key (human involvement) and an audit record are required.
Technical details
-
Agent Safety Architecture
The core principle is that all agent actions must pass through a proxy layer. This proxy stamps every outbound call with the agent's identity, preventing the agent from changing headers to bypass rate limits or ownership checks.
-
Proxy Functionality
The infrastructure layer (proxy) is responsible for enforcing deterministic rules (e.g., returning a 403 when a budget is crossed), regardless of the prompt's content or intent.
-
CI/CD Workloads
102s
The incident involved an agent deleting approximately 200 workloads in 90 seconds, impacting about 20 engineers. This highlights the risk of unconstrained cleanup actions.
Mentioned resources
- Anthropic
- CI team at Entropic
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.