# FinOps for AI Agents: Who Spent All the Tokens? — Tisha Chawla & Susheem Koul, Microsoft

## Executive summary

The talk introduces TokenOps, a control plane designed to shift AI agent development from 'token maxing' (spending tokens) to 'value maxing' (maximizing value per token). It addresses the critical gap in current systems: the lack of cost governance between code execution and model calls. TokenOps operates out-of-band at the entire agent run level, utilizing a `boundary annotation` and `governor node` to implement sophisticated policies that can 'steer' an agent's behavior (e.g., instructing it to be more succinct) before hitting a budget cap, thereby preventing costly failures.

## Key takeaways

- Shift from Token Maxing to Value Maxing: The industry needs to move beyond simply spending tokens and focus on ensuring that every token spent has measurable business value. This requires proper attribution of costs back to specific agent runs.
- Run-Level Cost Control is the Missing Piece: Existing tools (like model gateways) only control cost at the request level. TokenOps provides governance at the entire agent run layer, allowing control over complex loops and context growth.
- Steering vs. Halting: Instead of simply halting an agent when a budget is exceeded (a circuit breaker), the 'steer' action uses a cost guard to predict overruns and injects instructions into the system prompt, guiding the agent toward more efficient outputs.

## Technical details

- TokenOps Architecture: The platform is an out-of-band plane comprising three modules: Instrumentation (observability/telemetry), Accounting (maintaining a ledger of total runs and costs), and Enforcement. This design ensures it does not interfere with the existing agent code.
- Code Integration via Annotation: The `boundary annotation` is applied to methods within the agent's code, allowing the system to track inputs and outputs (floating attribution) without requiring a full code rewrite. This data feeds into the control plane.
- Control Plane Functionality: The control plane manages budgets against segments defined by emitted dimensions (e.g., `cohort=AI 2026`). It applies policies that can trigger either a hard 'halt' or a behavioral 'steer' action.
- Performance Benchmarks: Benchmarking on open-source repos showed that enabling the full policy suite reduced average token spend by approximately 78% and increased successful run completion rates from 67% to roughly 96%.

## Practical implications

- Allows build engineers to implement cost guardrails for production AI agents without modifying the core agent logic (out-of-band).
- Provides fine-grained control over resource consumption by segmenting budgets based on user dimensions or cohorts.
- Mitigates runaway costs in complex, multi-step agent workflows that traditional model gateways cannot track.

## Topics

FinOps, AI Agents, Cost Management, LLM Governance, Distributed Systems, Public Wiki

Source: https://www.youtube.com/watch?v=GJX19pNhmSw