Total Recall: Agent Memory and Harness Engineering — Ignacio Martinez, Oracle
Summary
The presentation details the architecture of an AI agent, arguing that while Large Language Models (LLMs) provide the non-deterministic 'reasoning core,' the true engineering effort lies in the 'agent harness.' This harness is a multi-layered system designed to transform the LLM's unpredictable output into reliable, repeatable, and autonomous workflows. Key architectural components include advanced memory management (short-term, long-term, shared), semantic layers for institutional knowledge capture, and robust data storage solutions that combine the flexibility of files with the transactional consistency of databases.
Key takeaways
-
The Agent Architecture Shift
17:30
An AI agent is defined as a Large Language Model (the reasoning core, which is rented and uncontrolled) augmented by a harness. The goal of harness engineering is to ensure reliable and predictable output, compensating for the model's inherent non-determinism.
-
Data Layer Complexity and Solutions
27:06
The data layer is critical, requiring a hybrid approach to storage. While files are simple and model-friendly, they lack transactional consistency. Databases provide ACID properties, high availability, and vector search. The optimal solution is a hybrid system, such as Oracle DBFS, which combines the benefits of both.
-
Memory and Context Management
33:20
Agent memory is categorized into short-term (ephemeral), long-term (episodic), and shared memory. To prevent 'context rot' and maintain relevance, the harness must implement sophisticated context engineering techniques, such as context compaction and summarization.
-
The Agent Workflow Loop
50:00
The core operational structure is the agent loop (Observe $ ightarrow$ Reason $ ightarrow$ Act). This loop must be failure-resistant and can be enhanced through techniques like the Toolbox pattern and model routing (using small experts for specific tasks) to improve efficiency and reliability.
Technical details
-
Agent Stack Components
720s
The agent stack consists of five layers: Application (user interface), Data (memory, knowledge, retrieval), Model (LLM), Infrastructure (orchestration), and Compute (cloud/GPUs, DB engine). The data layer is highlighted as the area of greatest control for engineers.
-
Data Storage Dilemma
1626s
The choice between files (easy, unstructured, POSIX semantics) and databases (structured, ACID, high availability) is addressed. The recommended solution is a hybrid approach, leveraging technologies like Oracle DBFS to provide both file-like accessibility and database-level transactional integrity.
-
Retrieval Augmented Generation (RAG)
2400s
The RAG process involves using a 'benccoder' (embedding model) to create vectors from documents, storing them in a vector store, and then using a cross-encoder (reranker) to compare the user query against the stored vectors to retrieve the most relevant context.
-
Semantic Layer
2800s
The semantic layer acts as the 'agent's lens' (drawing from Jakob von Uexküll's Umwelt), capturing proprietary or institutional knowledge—such as data modeling, query execution logic, or internal metadata—that is not explicitly provided to the LLM.
-
Agent Memory Types
2000s
Memory is segmented into short-term (ephemeral, e.g., current to-do list), long-term (episodic, e.g., past conversations), and shared memory (for multi-agent collaboration). The Oracle Agent Memory Package (OAMP) provides a managed solution for these components.
Mentioned resources
- GitHub Codespaces
- Oracle DBFS
- Oracle Agent Memory Package (OAMP)
- HNSW Index
- OCI Generative AI Service
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.