Memory Harnesses for Long-Running Research Agents — Stefania Druga, Sakana.ai
Summary
The presentation details the design and efficacy of 'memory harnesses' for managing state in long-running research agents. The core finding is that while memory harnesses offer no benefit when task context fits within the model's window (adding only cost), they are critical for solving long-horizon tasks where relevant information sits far outside the current context. The speaker emphasizes treating memory as a write-manage-read control loop, not merely an attached database.
Key takeaways
-
Memory is a Control Loop, Not a Database
5:44
The memory harness must be viewed as a 'write-manage-read' control loop wrapped around the model, rather than simply a retrieval database. This architectural focus allows for precise state management.
-
Local Models Enable Sovereignty and Cost Control
2:36
Running evaluations on local hardware (e.g., M3 Ultra) enables better control over the entire pipeline, which is crucial for maintaining 'sovereign AI' capabilities and reducing operational costs.
-
Ranked Recall Outperforms Other Policies
7:16
Across the X-Bench benchmark (over 68 questions), the 'ranked recall' policy consistently achieved the best performance, outperforming even approaches that simply gate memory usage or use an ideal 'oracle' ground truth.
Technical details
-
Memory Harness Architecture
411s
The proposed harness structure includes a core (showing traces to the agent), a recall block (testing different retrieval modes), and an archival block (tracking information across sessions). The tested policies include: no memory, vector RAG, decisions ledger (tracking prioritized decisions per turn), and an oracle (ground truth for optimal memory retrieval).
-
Long-Horizon Task Failure Modes
0s
In long tasks, agents suffer from 'context rot'—contradicting themselves or forgetting initial questions. The harness is necessary when the answer (e.g., at step 124) arrives far outside the context window relative to the question (at step 500).
-
Hardware and Model Setup
156s
Experiments were run on an M3 Ultra (96 GB, 28 core CPUs) using quantized models like Qwen 27B (4-bit) and Deep Seek V4 Flash.
Mentioned resources
- X-Bench
- Open-source repository (Cookbooks)
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.