Lessons from Studying Every Memory System — Shlok Khemani, Independent

Summary

The talk provides a deep dive into the evolution and architectural differences of memory systems in consumer AI applications (ChatGPT, Claude, Gemini). The core thesis is that 'memory' is not a standardized technology but rather a function of compute, requiring careful trade-offs between profile size, update frequency, and context window cost. Speakers highlight that while general architectures are converging toward running profiles, the specific implementation details remain unique to each product, meaning memory cannot be outsourced.

Technical details

• ChatGPT Memory Evolution 209s

  V1 (Feb 2024) was a user-managed list of facts, suffering from the burden of manual memory management and staleness. V2 (Apr 2025) introduced 'user knowledge memories'—a running profile updated asynchronously every few days, containing dense keyword clues (up to ~4,000 tokens).

• Claude Memory Architecture 723s

  V1 lacked a user profile, relying instead on two tools: searching previous conversations by keyword or time period. V2 added a running profile that is visible to the user, smaller (~1,000 tokens), uses complete sentences, and updates every 24 hours.

• The Running Profile Concept 408s

  A running profile synthesizes new conversation data into a persistent summary added to the context window. This process constitutes a continual learning loop that is currently happening outside of model weights.

• Memory System Flaws (Staleness/Conflict) 604s

  Even advanced systems can fail; ChatGPT's V2 profile showed overlapping dates for travel (Thailand and Turkey), demonstrating that the system cannot resolve conflicts or reason over external evidence (like flight bookings).

Mentioned resources

• Guardian Angels (Essay/Article)

Channel & topics

• AI Engineer
• AI Architecture
• LLM Memory Systems
• Product Design
• Continual Learning
• Context Management
• Guardian Angels