# Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle

## Executive summary

This session details the architecture of 'agent harnesses,' emphasizing that the scaffolding around a Large Language Model (LLM) is often more critical for real-world gains than the frontier model itself. The speaker outlines an agent stack designed for autonomous applications, using an Anti-Money Laundering (AML) use case for a mid-sized bank. Key components include the agent loop, advanced memory management (episodic, procedural, semantic), and sophisticated context engineering techniques like vectorization and hybrid search. The goal is to enable autonomous, continuous decision-making while managing the inherent limitations of context window size and model control.

## Key takeaways

- The Importance of the Agent Harness: The harness—comprising memory, tools, and perception—is the most customizable and impactful part of an AI agent system. It controls the agent's behavior, allowing the same LLM model to perform differently based on the configuration of the harness.
- Advanced Memory Architecture: Effective agents require multiple memory types: Short-term (context window), Episodic (past conversations/tool logs), Procedural (workflows/skills), and Semantic (user preferences/context). Managing this memory is crucial because the context window performance degrades as it grows.
- Context Management and Retrieval: To combat context window limitations, techniques like context summarization and compaction are used. Furthermore, tools and skills should be managed via vector embeddings and vector similarity search, often enhanced by a reranker, to keep the context relevant and limited.
- Data Layer Best Practices: For data storage, the speaker advises against the 'files vs. databases' dilemma. Databases are preferred due to their inherent support for transactional consistency (ACID consistency), which is necessary when multiple agents write data concurrently.

## Technical details

- Agentic Systems Architecture: An agentic system combines automation (LLM-driven workflows) and autonomy (AI agents). The core components of the data layer include the agent loop, a gateway/MCP for external data sources, the memory layer, and tools/skills.
- Memory Components: Memory is categorized into: Short-term (the immediate context window), Episodic (records of past events/conversations), Procedural (sequences of skills/workflows), and Semantic (implicit knowledge like user preferences).
- Context Optimization: Context window management relies on Context Summarization (generating a summary of past interactions) and Compaction (storing chunks of conversation in a database/file with an ID for later retrieval).
- Tool and Skill Management: To prevent context overflow from hundreds of tools/skills, they must be vectorized. The agent loop performs vector similarity search against a vector store to retrieve only the most relevant tools or skills for the current task. This process can be refined using a reranker.
- Continual Learning: A frozen model can improve over time through continual learning, which can be implemented via three memory types: Parametric (modifying model weights), Representational (using embeddings), or Nonparametric (the focus of the harness architecture).
- Data Retrieval and Search: Hybrid search, combining semantic search (embeddings) and keyword search, is recommended for superior retrieval quality. The combination is often implemented using RRF (Reciprocal Rank Fusion).
- Instruction Hierarchy: When interacting with a model via the Chat Completions API, the input has an inherent hierarchy. The System Prompt is the most important, followed by the User Prompt, and finally, the available tools and skills. This hierarchy allows the model to focus on specific instructions.

## Practical implications

- Build engineers should treat the agent harness as a modular, pluggable data layer, rather than trying to build a monolithic system.
- Focus on implementing robust data consistency (ACID) when designing data sources for agents, preferring structured databases over raw file systems.
- Design the agent loop to be highly configurable, allowing the maximum number of iterations and reasoning time to be adjusted based on the specific problem constraints.
- Implement a tiered memory strategy (short-term/long-term) and use vector databases to manage context, ensuring that only the most relevant information is passed to the LLM.

## Topics

AI Agents, Agent Harnesses, LLM Architecture, Memory Management, Vector Databases, Anti-Money Laundering (AML), workshopwaitingroom.com, GitHub, Oracle

Source: https://www.youtube.com/watch?v=KtBivPzmOM8