# Video Has No Memory. Here's How We Built One. — James Le, TwelveLabs

## Executive summary

Video AI systems currently lack 'memory,' meaning they process each query from scratch rather than maintaining a durable understanding of an entire video corpus. The solution proposed is building a dedicated memory layer by treating video not as a bag of frames, but as a spatial temporal volume. This architecture utilizes a Context Graph—a navigable structure connecting entities, appearances, relationships, and metadata across time—to enable deep reasoning, advanced tracking, and multi-source analysis (e.g., analyzing an entire World Cup corpus).

## Key takeaways

- Video is a Spatial Temporal Volume: Instead of treating video as a stack of images or text tokens, it must be modeled as a spatial temporal volume that preserves continuity across space, time, visual information, speech, sound, and motion. (0:50)
- The Need for a Memory Layer: Existing systems suffer from wrong context, weak memory (lacking durable continuity), and insufficient reasoning over complex video data. A dedicated memory layer is required to connect moments across different files or seasons. (2:06)
- Context Graph Structure: The optimal mental model for video intelligence is a Context Graph, which provides a durable, queryable representation linking time-bounded moments (evidence units), entity appearances, relationships, and corpus-level themes. (7:48)
- Core Design Principles: To build effective video memory, systems must 'ingest once and reason many times,' store primitives (not just answers), ground every claim to a specific timestamp, let intent shape the memory, and keep the layer composable via APIs. (9:04)

## Technical details

- Video Data Model: Video must be modeled as a spatial temporal volume, encompassing visual information, speech, sound, motion, OCR data, camera changes, and metadata, rather than just a sequence of frames. (0:50)
- The TwelveLabs Stack Architecture: The proposed stack includes semantic chunks capturing meaningful temporal units, Marengo (a multimodal embedding encoder) which converts these into spatial temporal relations, a Context Store for preserving reusable structure (moments, entities), and Pegasus (a Video Language Model/VLM) serving as the reasoning layer. (4:53)
- Context Graph Components: The graph is structured hierarchically: time-bounded moments $ ightarrow$ appearances (where/when an entity shows up) $ ightarrow$ entities (people, brands, places) $ ightarrow$ relationships (core occurrences) $ ightarrow$ corpus level context (main themes). (7:48)
- Video Worker Concept: Moving beyond static models, a 'video worker' operates within a deterministic system that can plan tasks, receive evidence, inspect relevant moments, synthesize results, and validate outputs against defined constraints. (10:45)

## Practical implications

- **Sports Analysis:** Tracking players (e.g., Messi) across an entire corpus and describing camera framing or identifying specific build-up plays (12:51).
- **Security/Surveillance:** Classifying vehicles, detecting safety events (e.g., near collisions), and analyzing crowded areas from public camera footage (15:37).
- **Advertising/Brand Safety:** Identifying high-impact moments within long-form content (e.g., a 5-minute Adidas clip) suitable for ad placement or brand integration (15:37).

## Topics

Video AI, Multimodal AI, Context Graph Theory, Natural Language Processing, Data Architecture, James Le, TwelveLabs

Source: https://www.youtube.com/watch?v=mOf-PP4mVjA