Prompt Caching Explained: Stop Overpaying for AI Agents

Published 2026-08-10 · Duration 17:16

Summary

Prompt caching is essential for managing costs in long-running AI agent sessions. Instead of paying full price for re-sending entire conversation histories (context windows) on every turn—which can lead to exponential cost increases—proper prompt caching ensures that the LLM only charges a discounted rate for tokens it has seen before. This requires designing an agent harness that correctly preserves reusable prompt prefixes and understands provider-specific API behaviors.

Download summary

Key takeaways

Cache Inputs, Not Outputs
Prompt caching stores and reuses inputs (the conversation history/prompt), not the LLM's outputs. Caching outputs is generally not useful for LLMs.
Cost Escalation Risk 0:23
Without caching, sending a growing context window (e.g., 51k tokens, then 55k tokens) repeatedly leads to exponentially increasing costs, making long sessions prohibitively expensive.
Cache Expiry is Critical 2:03
The cache has an expiration time (e.g., OpenAI OAuth API: one hour; Anthropic: five minutes by default). The agent harness must account for this expiry to avoid paying full price again.
System Prompts Must Be Static 8:00
To prevent cache invalidation, do not include dynamic elements like timestamps or current working directories within the system prompt. Keep conversation history append-only.

Technical details

Caching Mechanism 152s
Prompt caching works by having LLM providers charge a lower rate for tokens that have been processed previously in the session, routing processing to the same machine (cache hit).
API Differences 340s
Providers using the Responses API often enable caching automatically. However, when using Chat Completions APIs, developers must manually ensure caching is enabled.
Cost Modeling 258s
The cost structure involves a higher price for the first time an LLM sees a token and a significantly lower rate (e.g., 10% of the original price) for subsequent reads, which is the goal of caching.
Agent Frameworks 403s
Tools like Tau (a Python port of Pi) provide features to monitor cache hit rates and manage cached context windows within agent harnesses.

Mentioned resources

Written tutorial: Prompt Caching (Tutorial)
Tau coding agent (Agent Framework)
Pi coding agent (Agent Framework)
Hugging Face Inference Providers (Documentation)
OpenAI prompt caching (Documentation)
Anthropic prompt caching (Documentation)
Gemini context caching (Documentation)

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.