# Total Recall: Agent Memory and Harness Engineering — Ignacio Martinez, Oracle

## Executive summary

The presentation details the architecture of an AI agent, arguing that while Large Language Models (LLMs) provide the non-deterministic 'reasoning core,' the true engineering effort lies in the 'agent harness.' This harness is a multi-layered system designed to transform the LLM's unpredictable output into reliable, repeatable, and autonomous workflows. Key architectural components include advanced memory management (short-term, long-term, shared), semantic layers for institutional knowledge capture, and robust data storage solutions that combine the flexibility of files with the transactional consistency of databases.

## Key takeaways

- The Agent Architecture Shift: An AI agent is defined as a Large Language Model (the reasoning core, which is rented and uncontrolled) augmented by a harness. The goal of harness engineering is to ensure reliable and predictable output, compensating for the model's inherent non-determinism.
- Data Layer Complexity and Solutions: The data layer is critical, requiring a hybrid approach to storage. While files are simple and model-friendly, they lack transactional consistency. Databases provide ACID properties, high availability, and vector search. The optimal solution is a hybrid system, such as Oracle DBFS, which combines the benefits of both.
- Memory and Context Management: Agent memory is categorized into short-term (ephemeral), long-term (episodic), and shared memory. To prevent 'context rot' and maintain relevance, the harness must implement sophisticated context engineering techniques, such as context compaction and summarization.
- The Agent Workflow Loop: The core operational structure is the agent loop (Observe $ ightarrow$ Reason $ ightarrow$ Act). This loop must be failure-resistant and can be enhanced through techniques like the Toolbox pattern and model routing (using small experts for specific tasks) to improve efficiency and reliability.

## Technical details

- Agent Stack Components: The agent stack consists of five layers: Application (user interface), Data (memory, knowledge, retrieval), Model (LLM), Infrastructure (orchestration), and Compute (cloud/GPUs, DB engine). The data layer is highlighted as the area of greatest control for engineers.
- Data Storage Dilemma: The choice between files (easy, unstructured, POSIX semantics) and databases (structured, ACID, high availability) is addressed. The recommended solution is a hybrid approach, leveraging technologies like Oracle DBFS to provide both file-like accessibility and database-level transactional integrity.
- Retrieval Augmented Generation (RAG): The RAG process involves using a 'benccoder' (embedding model) to create vectors from documents, storing them in a vector store, and then using a cross-encoder (reranker) to compare the user query against the stored vectors to retrieve the most relevant context.
- Semantic Layer: The semantic layer acts as the 'agent's lens' (drawing from Jakob von Uexküll's Umwelt), capturing proprietary or institutional knowledge—such as data modeling, query execution logic, or internal metadata—that is not explicitly provided to the LLM.
- Agent Memory Types: Memory is segmented into short-term (ephemeral, e.g., current to-do list), long-term (episodic, e.g., past conversations), and shared memory (for multi-agent collaboration). The Oracle Agent Memory Package (OAMP) provides a managed solution for these components.

## Practical implications

- Architects should prioritize the data layer and memory components over the model itself, as this is where the most control and customization can be achieved.
- Implementing a hybrid storage solution (e.g., DBFS) is crucial for building enterprise-grade agents that require both file flexibility and transactional guarantees.
- Engineers must design for fault tolerance and autonomy by implementing robust agent loops that can recover from errors and manage tool calls.
- To improve efficiency, consider model routing, using specialized, smaller models (small experts) for specific tasks rather than relying solely on large, general-purpose LLMs.

## Topics

AI Agent Architecture, Harness Engineering, Vector Databases, Memory Management, RAG Systems, Workflow Automation, GitHub Codespaces, Oracle DBFS, Oracle Agent Memory Package (OAMP), HNSW Index, OCI Generative AI Service

Source: https://www.youtube.com/watch?v=xs-ob87TTzg