Prompt Caching Explained: Stop Overpaying for AI Agents
Prompt caching is essential for managing costs in long-running AI agent sessions. Instead of paying full price for re-sending entire conversation histories (context windows) on every turn—which can lead to exponential cost increases—proper prompt caching ensures that the LLM only charges a discounted rate for tokens it has seen before. This requires designing an agent harness that correctly preserves reusable prompt prefixes and understands provider-specific API behaviors.
Key takeaways
-
Cache Inputs, Not Outputs
Prompt caching stores and reuses inputs (the conversation history/prompt), not the LLM's outputs. Caching outputs is generally not useful for LLMs.
-
Cost Escalation Risk
0:23
Without caching, sending a growing context window (e.g., 51k tokens, then 55k tokens) repeatedly leads to exponentially increasing costs, making long sessions prohibitively expensive.
-
Cache Expiry is Critical
2:03
The cache has an expiration time (e.g., OpenAI OAuth API: one hour; Anthropic: five minutes by default). The agent harness must account for this expiry to avoid paying full price again.
-
System Prompts Must Be Static
8:00
To prevent cache invalidation, do not include dynamic elements like timestamps or current working directories within the system prompt. Keep conversation history append-only.