Stop Fine-Tuning to Fix Retrieval Problems — Anant Srivastava
The core engineering challenge in enterprise AI is not the model itself, but how knowledge reaches inference. The speaker argues that teams often build AI systems by accident, treating prompt, memory, and weights as a single 'ladder.' Instead, they are three distinct tools for three different jobs: Prompt for small, stable behavior; Memory for current, large, and citable knowledge; and Weights (fine-tuning) for reflexes that have stopped changing. The key to robust AI architecture is recognizing these boundaries and building a circulating system (harness) that manages the flow of knowledge between these three components.
Key takeaways
-
Prompt: Behavior, not Facts
9:19
The prompt should house small, stable, and editable instructions defining the agent's tone, persona, or behavior (e.g., offering human escalation after three failures). It should not be used to store facts, as this increases token cost and risks 'lost in the middle' problems.
-
Memory: Current, Large, and Citable
10:59
Memory (including external RAG and agent memory) is for knowledge that is current (changes faster than fine-tuning), large (cannot fit in the prompt), and citable (must point to a source). Access control (e.g., per-user scope) belongs here.
-
Weights: What Has Stopped Changing
18:43
Fine-tuning (weights) should only be used for reflexes or patterns that have stabilized and stopped changing (e.g., the format or structure of medical codes like ICD-10). Fine-tuning on facts (like runbooks or product catalogs) is often a retrieval problem.
-
The Circulating Architecture
A mature AI system is a loop: information moves from Memory to Prompt (context window) when a session starts, and patterns/formats move from Memory to Weights (fine-tuning) over time, allowing the agent to improve by doing its job.