Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle
Summary
This session details the architecture of 'agent harnesses,' emphasizing that the scaffolding around a Large Language Model (LLM) is often more critical for real-world gains than the frontier model itself. The speaker outlines an agent stack designed for autonomous applications, using an Anti-Money Laundering (AML) use case for a mid-sized bank. Key components include the agent loop, advanced memory management (episodic, procedural, semantic), and sophisticated context engineering techniques like vectorization and hybrid search. The goal is to enable autonomous, continuous decision-making while managing the inherent limitations of context window size and model control.
Key takeaways
-
The Importance of the Agent Harness
18:26
The harness—comprising memory, tools, and perception—is the most customizable and impactful part of an AI agent system. It controls the agent's behavior, allowing the same LLM model to perform differently based on the configuration of the harness.
-
Advanced Memory Architecture
25:56
Effective agents require multiple memory types: Short-term (context window), Episodic (past conversations/tool logs), Procedural (workflows/skills), and Semantic (user preferences/context). Managing this memory is crucial because the context window performance degrades as it grows.
-
Context Management and Retrieval
34:11
To combat context window limitations, techniques like context summarization and compaction are used. Furthermore, tools and skills should be managed via vector embeddings and vector similarity search, often enhanced by a reranker, to keep the context relevant and limited.
-
Data Layer Best Practices
For data storage, the speaker advises against the 'files vs. databases' dilemma. Databases are preferred due to their inherent support for transactional consistency (ACID consistency), which is necessary when multiple agents write data concurrently.
Technical details
-
Agentic Systems Architecture
831s
An agentic system combines automation (LLM-driven workflows) and autonomy (AI agents). The core components of the data layer include the agent loop, a gateway/MCP for external data sources, the memory layer, and tools/skills.
-
Memory Components
1556s
Memory is categorized into: Short-term (the immediate context window), Episodic (records of past events/conversations), Procedural (sequences of skills/workflows), and Semantic (implicit knowledge like user preferences).
-
Context Optimization
1901s
Context window management relies on Context Summarization (generating a summary of past interactions) and Compaction (storing chunks of conversation in a database/file with an ID for later retrieval).
-
Tool and Skill Management
2051s
To prevent context overflow from hundreds of tools/skills, they must be vectorized. The agent loop performs vector similarity search against a vector store to retrieve only the most relevant tools or skills for the current task. This process can be refined using a reranker.
-
Continual Learning
2415s
A frozen model can improve over time through continual learning, which can be implemented via three memory types: Parametric (modifying model weights), Representational (using embeddings), or Nonparametric (the focus of the harness architecture).
-
Data Retrieval and Search
Hybrid search, combining semantic search (embeddings) and keyword search, is recommended for superior retrieval quality. The combination is often implemented using RRF (Reciprocal Rank Fusion).
-
Instruction Hierarchy
When interacting with a model via the Chat Completions API, the input has an inherent hierarchy. The System Prompt is the most important, followed by the User Prompt, and finally, the available tools and skills. This hierarchy allows the model to focus on specific instructions.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.