FinOps for AI Agents: Who Spent All the Tokens? — Tisha Chawla & Susheem Koul, Microsoft
Summary
The talk introduces TokenOps, a control plane designed to shift AI agent development from 'token maxing' (spending tokens) to 'value maxing' (maximizing value per token). It addresses the critical gap in current systems: the lack of cost governance between code execution and model calls. TokenOps operates out-of-band at the entire agent run level, utilizing a `boundary annotation` and `governor node` to implement sophisticated policies that can 'steer' an agent's behavior (e.g., instructing it to be more succinct) before hitting a budget cap, thereby preventing costly failures.
Key takeaways
-
Shift from Token Maxing to Value Maxing
The industry needs to move beyond simply spending tokens and focus on ensuring that every token spent has measurable business value. This requires proper attribution of costs back to specific agent runs.
-
Run-Level Cost Control is the Missing Piece
2:38
Existing tools (like model gateways) only control cost at the request level. TokenOps provides governance at the entire agent run layer, allowing control over complex loops and context growth.
-
Steering vs. Halting
Instead of simply halting an agent when a budget is exceeded (a circuit breaker), the 'steer' action uses a cost guard to predict overruns and injects instructions into the system prompt, guiding the agent toward more efficient outputs.
Technical details
-
TokenOps Architecture
857s
The platform is an out-of-band plane comprising three modules: Instrumentation (observability/telemetry), Accounting (maintaining a ledger of total runs and costs), and Enforcement. This design ensures it does not interfere with the existing agent code.
-
Code Integration via Annotation
1025s
The `boundary annotation` is applied to methods within the agent's code, allowing the system to track inputs and outputs (floating attribution) without requiring a full code rewrite. This data feeds into the control plane.
-
Control Plane Functionality
The control plane manages budgets against segments defined by emitted dimensions (e.g., `cohort=AI 2026`). It applies policies that can trigger either a hard 'halt' or a behavioral 'steer' action.
-
Performance Benchmarks
Benchmarking on open-source repos showed that enabling the full policy suite reduced average token spend by approximately 78% and increased successful run completion rates from 67% to roughly 96%.
Mentioned resources
- Public Wiki
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.