Give the Agent a Budget, Not a Token — Sachin Malhotra, Anthropic
The presentation argues that granting AI agents 'unbounded power' via simple tokens is dangerous. Instead of narrowing the token scope (a boolean fix), engineers must implement a comprehensive 'budget' system with four dimensions: how much, how fast, what can be undone, and who notices. Key solutions include using asymmetric verbs (prioritizing loud failures), enforcing rate limits on writes, implementing trip wires over static allow lists, and utilizing an 'undo test' to size the overall safety policy.
Key takeaways
-
Budget vs. Token
7:03
A token is a boolean (yes/no) scope; a budget is multi-dimensional, considering volume, velocity, reversibility, and observability. The failure was giving the agent unbounded power, not the model itself.
-
Asymmetric Verbs
10:05
Prioritize granting agents access to operations that fail loudly (e.g., unskipping a test, which causes CI to go red) and keep critical failure verbs (like skipping a test) reserved for human intervention with an audit trail.
-
Rate Limits & Trip Wires
13:54
Implement rate limits on every write operation, ensuring the ceiling refills automatically. Use trip wires (monitoring aggregate behavior) instead of static allow lists, as trip wires adapt to real-world data.
-
The Undo Test
This test asks if the agent can autonomously roll back its own changes and what the blast radius would be if it failed. If not, a second key (human involvement) and an audit record are required.
