# Tokens Should Have Jobs — Katelyn Lesse & Angela Jiang, Anthropic

## Executive summary

The presentation challenges the assumption that all tokens used in agentic systems are fungible. Instead of relying solely on increasing the token budget (brute force), the speakers propose giving tokens specific 'jobs' through advanced strategies. These strategies—Advising, Grading, and Dreaming—allow agents to improve performance and reliability within a fixed budget, particularly when the task requires high precision (e.g., financial analysis). The analysis showed that while increasing the budget improves performance across all methods, specialized strategies significantly reduce the true cost (tokens) required to achieve a perfect, reliable outcome.

## Key takeaways

- Tokens are not fungible: The core thesis is that tokens should be assigned specific jobs (e.g., advising, grading, dreaming) rather than being used indiscriminately for simple execution. This allows for more efficient use of computational resources.
- Three core agent strategies: Strategies include: 1) Advising (splitting an executor from an adviser that provides mid-task guidance); 2) Grading (defining a rubric and using a grader to iterate until a perfect outcome is achieved); and 3) Dreaming (allowing tokens to inspect transcripts and write findings to memory for future runs).
- Cost vs. Accuracy in Financial Analysis: When evaluating financial tasks (like creating a P&L statement), 80% accuracy is considered useless if 100% accuracy is required. The analysis showed that the baseline 'Execute' strategy required an average of 1.8 million tokens to achieve a perfect answer, while 'Advise' and 'Grade' were significantly more token-efficient.
- Optimization depends on the goal: The choice of strategy depends on the optimization goal: choose 'Advise' for maximum token efficiency, or 'Grade'/'Dream' if maximizing the percentage of perfect, reliable runs is the priority.

## Technical details

- Agentic Strategies: The three proposed strategies are: Advising (Executor calls out to an Adviser for real-time guidance); Grading (Uses a predefined rubric and a Grader to enforce iterative improvement); and Dreaming (The Dreamer inspects the Executor's transcripts and writes findings to memory for subsequent runs).
- System Architecture: The system utilizes a 'meta-harness' level for multi-agent orchestration, coordinating primitives between the executor, adviser, and grader. This allows for complex, dynamic setups.
- Benchmarking: The experiments were conducted on a financial analysis bench, simulating expert human performance. The baseline 'Execute' strategy was compared against the advanced strategies using a fixed budget (e.g., 600,000 tokens) to isolate the effect of the job type.

## Practical implications

- When building complex agents, do not assume tokens are fungible; instead, design specific roles (adviser, grader, dreamer) for tokens to maximize efficiency.
- For critical domains (like finance), prioritize reliability (100% accuracy) over raw accuracy percentage, as partial results are often useless.
- Implement multi-agent orchestration layers (meta-harnesses) to combine different strategies and build complex, dynamic problem-solving architectures.

## Topics

Agentic Systems, LLMs, Prompt Engineering, AI Architecture, Token Optimization, Anthropic, Katelyn Lesse, Angela Jiang

Source: https://www.youtube.com/watch?v=PXj0p_mW9nI