Video Has No Memory. Here's How We Built One. — James Le, TwelveLabs
Video AI systems currently lack 'memory,' meaning they process each query from scratch rather than maintaining a durable understanding of an entire video corpus. The solution proposed is building a dedicated memory layer by treating video not as a bag of frames, but as a spatial temporal volume. This architecture utilizes a Context Graph—a navigable structure connecting entities, appearances, relationships, and metadata across time—to enable deep reasoning, advanced tracking, and multi-source analysis (e.g., analyzing an entire World Cup corpus).
Key takeaways
-
Video is a Spatial Temporal Volume
0:50
Instead of treating video as a stack of images or text tokens, it must be modeled as a spatial temporal volume that preserves continuity across space, time, visual information, speech, sound, and motion. (0:50)
-
The Need for a Memory Layer
2:06
Existing systems suffer from wrong context, weak memory (lacking durable continuity), and insufficient reasoning over complex video data. A dedicated memory layer is required to connect moments across different files or seasons. (2:06)
-
Context Graph Structure
7:48
The optimal mental model for video intelligence is a Context Graph, which provides a durable, queryable representation linking time-bounded moments (evidence units), entity appearances, relationships, and corpus-level themes. (7:48)
-
Core Design Principles
9:04
To build effective video memory, systems must 'ingest once and reason many times,' store primitives (not just answers), ground every claim to a specific timestamp, let intent shape the memory, and keep the layer composable via APIs. (9:04)