Tokens Should Have Jobs — Katelyn Lesse & Angela Jiang, Anthropic
Summary
The presentation challenges the assumption that all tokens used in agentic systems are fungible. Instead of relying solely on increasing the token budget (brute force), the speakers propose giving tokens specific 'jobs' through advanced strategies. These strategies—Advising, Grading, and Dreaming—allow agents to improve performance and reliability within a fixed budget, particularly when the task requires high precision (e.g., financial analysis). The analysis showed that while increasing the budget improves performance across all methods, specialized strategies significantly reduce the true cost (tokens) required to achieve a perfect, reliable outcome.
Key takeaways
-
Tokens are not fungible
The core thesis is that tokens should be assigned specific jobs (e.g., advising, grading, dreaming) rather than being used indiscriminately for simple execution. This allows for more efficient use of computational resources.
-
Three core agent strategies
3:45
Strategies include: 1) Advising (splitting an executor from an adviser that provides mid-task guidance); 2) Grading (defining a rubric and using a grader to iterate until a perfect outcome is achieved); and 3) Dreaming (allowing tokens to inspect transcripts and write findings to memory for future runs).
-
Cost vs. Accuracy in Financial Analysis
11:40
When evaluating financial tasks (like creating a P&L statement), 80% accuracy is considered useless if 100% accuracy is required. The analysis showed that the baseline 'Execute' strategy required an average of 1.8 million tokens to achieve a perfect answer, while 'Advise' and 'Grade' were significantly more token-efficient.
-
Optimization depends on the goal
The choice of strategy depends on the optimization goal: choose 'Advise' for maximum token efficiency, or 'Grade'/'Dream' if maximizing the percentage of perfect, reliable runs is the priority.
Technical details
-
Agentic Strategies
225s
The three proposed strategies are: Advising (Executor calls out to an Adviser for real-time guidance); Grading (Uses a predefined rubric and a Grader to enforce iterative improvement); and Dreaming (The Dreamer inspects the Executor's transcripts and writes findings to memory for subsequent runs).
-
System Architecture
The system utilizes a 'meta-harness' level for multi-agent orchestration, coordinating primitives between the executor, adviser, and grader. This allows for complex, dynamic setups.
-
Benchmarking
428s
The experiments were conducted on a financial analysis bench, simulating expert human performance. The baseline 'Execute' strategy was compared against the advanced strategies using a fixed budget (e.g., 600,000 tokens) to isolate the effect of the job type.
Mentioned resources
- Anthropic
- Katelyn Lesse
- Angela Jiang
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.