Natural Language Processing — The Video Signal

Aug 7, 2026 · 16:25

How Harmonic 4x'd User Retention by Building on Deep Agents

Harmonic transitioned its natural language interface, Scout, from a brittle query parsing graph to an architecture built on Deep Agents and a simple model-plus-tools loop. This shift quadrupled week one to week four user retention. The core technical lesson is that robust agent design requires managing context via a 'harness contract,' ensuring that all artifacts (like visualizations or large search result sets) are visible to the model—either in the message list or offloaded through file system tools—to prevent the UX from becoming an invisible black box.

Key takeaways

Deep Agents significantly boost retention 2:04
Switching to Deep Agents resulted in a fourfold increase in week one to week four user retention for Scout. (1:24)
The agent architecture simplified from graphs to loops 4:01
Scout evolved from complex, multi-node query parsing graphs (LangGraph) into a simpler model and tools loop, mediated by middleware. (2:41)
Context management is handled by the harness 8:16
Deep Agents manage context overload using mechanisms like compaction for long message lists and file system abstraction to store large results, returning only pointers to the model. (4:56)
UX must respect the agent's context contract 11:44
For a product UX to be useful, any rendered element (e.g., charts) must either reside in the message list or be discoverable by the model via tools/file system pointers; otherwise, it is invisible to the agent. (7:04)

Watch on YouTube Full article

Jul 28, 2026 · 46:14

Build Smarter Voice Agents

This panel discussion provides deep insights into building production-grade voice AI agents. Key architectural recommendations favor the deconstructed cascading pipeline (ASR $\rightarrow$ LLM $\rightarrow$ TTS) due to its superior flexibility for optimization and model swapping. Engineers must prioritize managing latency within a 1–1.5 second budget, implementing robust fallback systems across all stack components (ASR, LLM, TTS), and utilizing advanced context management techniques like 'Scratchpads' to maintain conversational continuity over long interactions.

Key takeaways

Architectural Choice: Cascading Pipeline 1:45
The cascading architecture is preferred because it allows for individual optimization of the ASR, LLM, and TTS layers. This modularity provides greater flexibility than a full Speech-to-Speech (S2S) stack when integrating new models or optimizing specific components.
Latency Management 4:23
The 'golden metric' for voice agent response time is between 1 to 1.5 seconds. Exceeding this budget can be unnerving for users, making latency a primary design constraint over pure accuracy in many cases.
Context and Memory Management 18:05
To prevent negative user sentiment from repeating information, agents must implement context stores (e.g., 'Scratchpads') to track customer profiles, preferences, and key facts across multiple turns or sessions.

Watch on YouTube Full article

Jul 23, 2026 · 20:27

Video Has No Memory. Here's How We Built One. — James Le, TwelveLabs

Video AI systems currently lack 'memory,' meaning they process each query from scratch rather than maintaining a durable understanding of an entire video corpus. The solution proposed is building a dedicated memory layer by treating video not as a bag of frames, but as a spatial temporal volume. This architecture utilizes a Context Graph—a navigable structure connecting entities, appearances, relationships, and metadata across time—to enable deep reasoning, advanced tracking, and multi-source analysis (e.g., analyzing an entire World Cup corpus).

Key takeaways

Video is a Spatial Temporal Volume 0:50
Instead of treating video as a stack of images or text tokens, it must be modeled as a spatial temporal volume that preserves continuity across space, time, visual information, speech, sound, and motion. (0:50)
The Need for a Memory Layer 2:06
Existing systems suffer from wrong context, weak memory (lacking durable continuity), and insufficient reasoning over complex video data. A dedicated memory layer is required to connect moments across different files or seasons. (2:06)
Context Graph Structure 7:48
The optimal mental model for video intelligence is a Context Graph, which provides a durable, queryable representation linking time-bounded moments (evidence units), entity appearances, relationships, and corpus-level themes. (7:48)
Core Design Principles 9:04
To build effective video memory, systems must 'ingest once and reason many times,' store primitives (not just answers), ground every claim to a specific timestamp, let intent shape the memory, and keep the layer composable via APIs. (9:04)

Watch on YouTube Full article

Jul 1, 2026 · 10:07

Universal 3.5 Pro Demo: Smarter Speech-to-Text with Contextual Awareness

This demo introduces Universal 3.5 Pro, an advanced Speech-to-Text (STT) model designed to significantly boost transcription accuracy through enhanced contextual awareness. Key features include passing domain-specific prompts (e.g., 'cardiology consultation'), applying context to key terms to prevent misapplication, and supporting dynamic mid-call prompt updates via API calls. Furthermore, the model retains conversation history (agent context), allowing it to accurately transcribe user input even in poor audio conditions by understanding the situational flow of a voice agent interaction.

Key takeaways

Contextual Prompting for Domain Accuracy
Passing detailed information about the audio content (e.g., 'cardiology consultation between Dr. Smith and elderly patient regarding chest pain...') dramatically improves model accuracy within specific domains. The more specific the prompt, the better the results.
Contextual Key Terms 2:00
Unlike previous methods where key terms were applied blindly, Universal 3.5 Pro allows users to define what a key term represents (e.g., 'The user's name is Zachary Klebanoff'). This prevents the model from incorrectly applying terminology based solely on acoustic similarity.
Dynamic Mid-Call Prompt Updates 2:55
The prompt can be updated in real time via the API (not available in the playground demo). This is crucial for voice agents, allowing tool calls or external data to adjust the model's context mid-conversation.
Conversation/Agent Context 3:30
The model retains previous transcriptions and accepts LLM-generated responses from a voice agent as context. This provides situational awareness, improving accuracy even in poor audio conditions and reducing the Word Error Rate (WER) on voice agent datasets.

Watch on YouTube Full article