# Prompt Caching Explained: Stop Overpaying for AI Agents

## Executive summary

Prompt caching is essential for managing costs in long-running AI agent sessions. Instead of paying full price for re-sending entire conversation histories (context windows) on every turn—which can lead to exponential cost increases—proper prompt caching ensures that the LLM only charges a discounted rate for tokens it has seen before. This requires designing an agent harness that correctly preserves reusable prompt prefixes and understands provider-specific API behaviors.

## Key takeaways

- Cache Inputs, Not Outputs: Prompt caching stores and reuses inputs (the conversation history/prompt), not the LLM's outputs. Caching outputs is generally not useful for LLMs.
- Cost Escalation Risk: Without caching, sending a growing context window (e.g., 51k tokens, then 55k tokens) repeatedly leads to exponentially increasing costs, making long sessions prohibitively expensive.
- Cache Expiry is Critical: The cache has an expiration time (e.g., OpenAI OAuth API: one hour; Anthropic: five minutes by default). The agent harness must account for this expiry to avoid paying full price again.
- System Prompts Must Be Static: To prevent cache invalidation, do not include dynamic elements like timestamps or current working directories within the system prompt. Keep conversation history append-only.

## Technical details

- Caching Mechanism: Prompt caching works by having LLM providers charge a lower rate for tokens that have been processed previously in the session, routing processing to the same machine (cache hit).
- API Differences: Providers using the Responses API often enable caching automatically. However, when using Chat Completions APIs, developers must manually ensure caching is enabled.
- Cost Modeling: The cost structure involves a higher price for the first time an LLM sees a token and a significantly lower rate (e.g., 10% of the original price) for subsequent reads, which is the goal of caching.
- Agent Frameworks: Tools like Tau (a Python port of Pi) provide features to monitor cache hit rates and manage cached context windows within agent harnesses.

## Practical implications

- Implement an agent harness that explicitly monitors and utilizes cache hit rates.
- Ensure the system prompt remains static (no dynamic variables like timestamps or directories) to prevent cache invalidation.
- Design conversation history to be append-only, avoiding unnecessary structural changes.
- Be aware of the specific cache expiry time for every LLM provider used in the build pipeline.

## Topics

AI Agents, LLM Architecture, Cost Optimization, Prompt Engineering, Software Design, Written tutorial: Prompt Caching, Tau coding agent, Pi coding agent, Hugging Face Inference Providers, OpenAI prompt caching, Anthropic prompt caching, Gemini context caching

Source: https://www.youtube.com/watch?v=SkM4k4SKvCM