Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI
Summary
This talk provides a deep dive into context engineering for large language model (LLM) agents, focusing on optimizing performance and cost in real-world applications like an AI tutor. The core finding is that compaction techniques (like summarization) are often detrimental because they invalidate the prompt cache, forcing the system to pay full price for every token. For optimal recall and cost efficiency, retaining the full history—especially when leveraging advanced caching mechanisms—is superior to aggressive context reduction.
Key takeaways
-
Compaction is a potential trap due to Prompt Caching
22:06
When using prompt caching (which can make cached tokens up to 50 times cheaper), any transformation or summarization of the context invalidates the cache, forcing full-price token usage. Compaction must shrink the context by more than 50 times just to pay for itself.
-
Full History Retention Wins on Recall
33:33
Experiments showed that leaving the full history untouched (the 'full history' preset) provided the best memory recall and overall performance, even though it was the most expensive option. Aggressive compaction techniques significantly degraded factual retrieval.
-
Hybrid Search is Superior for Retrieval
22:48
For knowledge base browsing, a hybrid search combining semantic similarity (embedding model) with keyword search (BM25) is necessary to achieve high recall. Pure dense retrieval failed when facts were buried at large context sizes (e.g., 400k tokens), while BM25 maintained 100% accuracy.
Technical details
-
Context Management Components
710s
An agent's context includes the system prompt, course content chunks, tool definitions, chat history, old tool outputs (tool calls/results pairs), and user questions. Optimizing this entire input stream is key.
-
Context Window Limitations
432s
The model's context window is finite. The primary challenges are managing context within a single session and maintaining memory across multiple sessions (statelessness).
-
Compaction Techniques
1508s
Methods include selective retention (LLMs deciding what to keep), continuous summarization, sliding window trimming (using the last N turns), and progressive disclosure (loading only necessary skills/context).
-
Retrieval Augmented Generation (RAG)
1368s
The system uses a hybrid search pipeline: Semantic Search (e.g., using Coherent model) combined with BM25 for keyword indexing. The knowledge base is structured into 'raw' files, 'generated' titles, and a 'wiki' index.
Mentioned resources
- Hugging Face Space
- AI Tutor App Repository
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.