The Video Signal

Topic

Natural Language Processing

All digests tagged Natural Language Processing

Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex thumbnail

· 21:04

Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex

The talk outlines the evolution of Retrieval-Augmented Generation (RAG) toward 2026, positing that RAG will decompose into an agent harness and a dedicated context layer. The primary challenge remains unlocking the vast, unstructured context locked in enterprise documents (PDFs, Word, etc.). The solution requires a three-layered platform: 1) Parsing into token-efficient markdown; 2) Semantic storage/document management; and 3) Repeatable, specialized workflows (e.g., KYC, invoice processing). The speaker emphasizes that document OCR is a complex problem due to how PDFs are rendered for display, not machine consumption, necessitating hybrid parsing approaches that combine deep binary understanding with advanced Vision-Language Models (VLMs).

Key takeaways

  1. RAG in 2026: Agent Harness + Context Layer

    Naive RAG (chunking, embedding, top-k retrieval) is outdated. Modern agents integrate retrieval complexity into the agent layer, allowing them to reason about the optimal search term rather than relying on fixed top-k retrieval methods. (0:00 - 1:33)

  2. Context Moves Up the Stack 5:11

    The focus is shifting from managing context window overflow to hooking up specialized Micro-Capabilities Platforms (MCP) and skills. Furthermore, defining programs and tasks is moving from code (Python, TypeScript) toward natural English language definitions. (3:11)

  3. Document Parsing Requires Hybrid Approaches 8:35

    Because PDFs are designed for printing (using glyphs and line segments) and not machine reading, document OCR is difficult. The optimal approach combines pipeline-based understanding of file binaries (e.g., Word's custom XML) with VLM-based visual analysis to achieve high accuracy and low cost. (5:15)

  4. The Three-Layer Document Platform 8:35

    A comprehensive document context layer requires three components: 1) Parsing into token-efficient markdown/metadata; 2) Semantic storage (document management for agents); and 3) Repeatable, specialized workflows (e.g., claims, invoices). (5:15)

Watch on YouTube Full article

If we want them to do Knowledge Work, design them as Knowledge Agents — Benjamin Clavié, Mixedbread thumbnail

· 17:55

If we want them to do Knowledge Work, design them as Knowledge Agents — Benjamin Clavié, Mixedbread

The talk distinguishes between 'Coding Agents' and 'Knowledge Agents,' arguing that most real-world tasks fall under the latter. While code provides durable cues (identifiers, file paths), knowledge work is inherently ambiguous, diffuse, and context-dependent (e.g., legal or medical research). The speaker posits that AI agents must be designed to mimic human knowledge work patterns—specifically, through advanced orchestration. This involves breaking down complex, open-ended problems, utilizing multiple specialized tools (primitives like BM25 and semantic search), and employing sub-agents (searchers) to synthesize findings into memos, thereby reducing the 'oracle gap' between perfect knowledge and the agent's output.

Key takeaways

  1. Knowledge Work vs. Coding Work

    Coding is a special, easy case because code has durable cues (identifiers, method definitions). Knowledge work, however, is defined by ambiguous information input and requires reconstructing intent and judgment, making it significantly harder for agents.

  2. The Knowledge Loop 9:15

    Human progress in knowledge work is driven by a single self-optimizing loop: better tools create new roles, new roles generate knowledge, and new knowledge demands better tools. This pattern should guide agent design.

  3. The Necessity of Orchestration

    Effective agent performance requires more than just a single tool. The most significant gains come from architectural improvements, such as having a main agent delegate tasks to specialized 'searcher agents' that return structured memos, which reduces the 'oracle gap' (the difference between perfect information and the system's output) by up to 40%.

  4. Tooling is not Neutral

    Tools are not merely incremental improvements; they are critical for overcoming performance ceilings. The ability to use a tool (e.g., a library catalog vs. physically searching archives) determines if a task is scalable and cheap enough to be practical.

Watch on YouTube Full article

Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI thumbnail

· 13:14

Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI

This presentation details the architecture of a real-time voice agent harness designed to achieve Frontier-level intelligence while maintaining low latency. The system utilizes a cascaded voice stack, drawing parallels to self-driving car systems, breaking the process into Perception (Transcription), Planning (LLM/Tool Calling), and Control (Speech Synthesis). Key innovations include a streaming speculative transcriber for accuracy, background agents for tool calling, and a prefix cache combined with audio suppression techniques to hide generation latency and ensure seamless, natural conversation flow.

Key takeaways

  1. Cascaded Voice Agent Architecture

    The system is structured into three layers: Perception (Transcription, converting audio to data), Planning (LLM, processing data and determining actions), and Control (Speech Synthesis, converting text back to natural audio).

  2. Streaming Speculative Transcriber 2:32

    A hybrid approach combining a fast streaming transcriber (e.g., Flux) with a slower, more accurate batch transcription (e.g., Scribe V2) that uses context to correctly identify entities like names and dates of birth.

  3. Background Tool Calling 7:04

    To reduce round trips with slow, intelligent LLMs, background agents perform tool calls and inject the results into the main model's context, making the main agent believe it executed the call itself.

  4. Prefix Cache for Synthesis 10:57

    The prefix cache monitors the model's output stream, checking if audio for a sequence of words already exists from a prior turn. This allows the agent to start speaking immediately from cached audio while the rest of the sentence is being generated.

Watch on YouTube Full article

Tolan: Voice-First AI Companion — Paula Dozsa, Tolan thumbnail

· 15:08

Tolan: Voice-First AI Companion — Paula Dozsa, Tolan

Paula Dozsa, an engineer on the Tolan team, details the unique engineering challenges of building a voice-first AI companion compared to traditional text-based LLM applications. She emphasizes that voice introduces 'conversational volatility' (fast turns, interruptions) which requires fundamental shifts in pipeline design, including smart turn detection, tiered model routing based on emotional stakes, and rebuilding context per turn rather than relying on a continuous cache. The talk also covers using advanced AI agents (like Claude) to build the product itself, achieving significant improvements in stability and feature development.

Key takeaways

  1. Voice vs. Text LLM Assumptions 4:05

    Text chat assumes slow turns and stable context, while voice is characterized by fast turns and volatile context (interruptions, subject changes mid-sentence). This volatility requires building for messy, real-world speech patterns.

  2. Optimizing for Interruptions 5:38

    Instead of minimizing interruptions, the team focused on building smart turn-taking that reads speech patterns to prevent early, incorrect agent interventions. They paid an extra 60 milliseconds of latency to achieve this.

  3. Memory as Retrieval System 12:32

    To handle volatile context, memory is treated as a retrieval system, not a transcript. Facts and preferences are embedded, stored in a vector database (with sub-50ms lookups), and compressed nightly to resolve contradictions and merge duplicates.

  4. Context Reassembly

    Context must be reassembled from parts every single turn (summary, user persona, retrieved memories, tone guidance) because reusing old context is a 'trap' when the user pivots subjects.

Watch on YouTube Full article

Score Every Production Trace with an LLM Judge, from Your Terminal (LangSmith CLI) thumbnail

· 5:08

Score Every Production Trace with an LLM Judge, from Your Terminal (LangSmith CLI)

This walkthrough demonstrates how to implement automated, large-scale evaluation of AI agents using an LLM-as-a-judge within LangSmith. By setting up an online evaluation, developers can programmatically score incoming chat traces (e.g., checking for user frustration) without manually reviewing thousands of interactions. The process involves using a coding agent to install LangSmith skills and configure the evaluation, followed by managing the sampling rate to control costs.

Key takeaways

  1. Online LLM-as-a-Judge Setup

    Online evaluations allow an LLM to automatically score incoming agent traces based on a defined prompt and rubric (e.g., detecting user frustration). The output is a score and reasoning attached directly to the trace.

  2. Using LangSmith CLI for Automation 0:01

    A coding agent can be instructed to use the LangSmith CLI to create and manage the LLM judge evaluator, automating the setup process.

  3. Cost Management via Sampling Rate 0:04

    The sampling rate can be updated (e.g., from 100% to 50%) to reduce evaluation costs while maintaining continuous monitoring.

Watch on YouTube Full article

Workshop: Building and optimizing dictation features thumbnail

· 36:06

Workshop: Building and optimizing dictation features

This workshop details the architecture for building low-latency dictation features using AssemblyAI's APIs. The core strategy involves leveraging the Sync API—a single POST request—to achieve speed superior to traditional async or streaming methods. The process is optimized through three key stages: using key terms prompting for accuracy, implementing connection warming to minimize network overhead (DNS, TCP, TLS), and running a cleanup pass via LLM Gateway (e.g., Qwen3.5 4B Fast) to refine raw transcripts into polished, intent-preserving text. The goal is to deliver results on screen in under one second.

Key takeaways

  1. Sync API Preference 5:35

    The Sync API is preferred for dictation because it avoids the overhead of maintaining a WebSocket connection (streaming) and bypasses the inherent latency floor of traditional async endpoints, which is unsuitable for short, burst dictation sessions.

  2. Latency Optimization via Warming 11:40

    To minimize latency, it is critical to call a warm endpoint (`client.sync.warm`) while the user is recording. This pre-pays the networking costs (DNS, TCP, TLS handshake), ensuring the subsequent transcription request goes straight to inference.

  3. Cleanup Pass with LLM Gateway 18:20

    A cleanup step using an LLM (like Qwen3.5 4B Fast) is necessary to transform raw, spoken text (e.g., 'I think we should meet in 5 minutes') into polished, corrected text while preserving the original intent. This is achieved by prompting the model to act as a function in a pipeline, not an assistant.

  4. Upcoming Dictation API 30:00

    AssemblyAI is rolling out a dedicated Dictation API (`client.dictation.describe`) that will wrap the entire loop (STT + Cleanup) into a single, simplified call, making the process easier for developers.

Watch on YouTube Full article

Tell the Robot What You Want — Sandhya Subramani, AWS thumbnail

· 17:23

Tell the Robot What You Want — Sandhya Subramani, AWS

The presentation demonstrates 'Scout,' a rover running on a Raspberry Pi, that has been upgraded from a traditionally programmed robot into an agentic system using AWS's open-source framework, `strands`. This architecture allows the robot to understand and execute complex commands in natural language (e.g., counting people or responding to arbitrary prompts) by having an AI agent orchestrate calls to existing hardware policies/tools. The system utilizes a multi-agent setup (thinker, communication, voice) and operates on a hybrid cloud/edge model for both training (VLA models) and fast runtime execution.

Key takeaways

  1. Agentic Robotics Architecture 2:07

    By implementing an agent layer above existing movement policies, the robot can perform tasks it was never explicitly trained for. The agent decides *what* to do, while the policy determines *how* to execute the action.

  2. Multi-Agent System Design 5:46

    Scout runs three simultaneous agents: a thinker (constantly assessing the environment), a communication agent (via Telegram/web app), and an optional voice agent, allowing for complex, concurrent decision-making.

  3. Hybrid Cloud/Edge Deployment 11:57

    The system is designed to operate on a hybrid model: training (VLA models) occurs in the cloud using `agentcore`, while execution and policy calls happen rapidly at the edge (on the robot's local hardware).

Watch on YouTube Full article

How Harmonic 4x'd User Retention by Building on Deep Agents thumbnail

· 16:25

How Harmonic 4x'd User Retention by Building on Deep Agents

Harmonic transitioned its natural language interface, Scout, from a brittle query parsing graph to an architecture built on Deep Agents and a simple model-plus-tools loop. This shift quadrupled week one to week four user retention. The core technical lesson is that robust agent design requires managing context via a 'harness contract,' ensuring that all artifacts (like visualizations or large search result sets) are visible to the model—either in the message list or offloaded through file system tools—to prevent the UX from becoming an invisible black box.

Key takeaways

  1. Deep Agents significantly boost retention 2:04

    Switching to Deep Agents resulted in a fourfold increase in week one to week four user retention for Scout. (1:24)

  2. The agent architecture simplified from graphs to loops 4:01

    Scout evolved from complex, multi-node query parsing graphs (LangGraph) into a simpler model and tools loop, mediated by middleware. (2:41)

  3. Context management is handled by the harness 8:16

    Deep Agents manage context overload using mechanisms like compaction for long message lists and file system abstraction to store large results, returning only pointers to the model. (4:56)

  4. UX must respect the agent's context contract 11:44

    For a product UX to be useful, any rendered element (e.g., charts) must either reside in the message list or be discoverable by the model via tools/file system pointers; otherwise, it is invisible to the agent. (7:04)

Watch on YouTube Full article

Build Smarter Voice Agents thumbnail

· 46:14

Build Smarter Voice Agents

This panel discussion provides deep insights into building production-grade voice AI agents. Key architectural recommendations favor the deconstructed cascading pipeline (ASR $\rightarrow$ LLM $\rightarrow$ TTS) due to its superior flexibility for optimization and model swapping. Engineers must prioritize managing latency within a 1–1.5 second budget, implementing robust fallback systems across all stack components (ASR, LLM, TTS), and utilizing advanced context management techniques like 'Scratchpads' to maintain conversational continuity over long interactions.

Key takeaways

  1. Architectural Choice: Cascading Pipeline 1:45

    The cascading architecture is preferred because it allows for individual optimization of the ASR, LLM, and TTS layers. This modularity provides greater flexibility than a full Speech-to-Speech (S2S) stack when integrating new models or optimizing specific components.

  2. Latency Management 4:23

    The 'golden metric' for voice agent response time is between 1 to 1.5 seconds. Exceeding this budget can be unnerving for users, making latency a primary design constraint over pure accuracy in many cases.

  3. Context and Memory Management 18:05

    To prevent negative user sentiment from repeating information, agents must implement context stores (e.g., 'Scratchpads') to track customer profiles, preferences, and key facts across multiple turns or sessions.

Watch on YouTube Full article

Video Has No Memory. Here's How We Built One. — James Le, TwelveLabs thumbnail

· 20:27

Video Has No Memory. Here's How We Built One. — James Le, TwelveLabs

Video AI systems currently lack 'memory,' meaning they process each query from scratch rather than maintaining a durable understanding of an entire video corpus. The solution proposed is building a dedicated memory layer by treating video not as a bag of frames, but as a spatial temporal volume. This architecture utilizes a Context Graph—a navigable structure connecting entities, appearances, relationships, and metadata across time—to enable deep reasoning, advanced tracking, and multi-source analysis (e.g., analyzing an entire World Cup corpus).

Key takeaways

  1. Video is a Spatial Temporal Volume 0:50

    Instead of treating video as a stack of images or text tokens, it must be modeled as a spatial temporal volume that preserves continuity across space, time, visual information, speech, sound, and motion. (0:50)

  2. The Need for a Memory Layer 2:06

    Existing systems suffer from wrong context, weak memory (lacking durable continuity), and insufficient reasoning over complex video data. A dedicated memory layer is required to connect moments across different files or seasons. (2:06)

  3. Context Graph Structure 7:48

    The optimal mental model for video intelligence is a Context Graph, which provides a durable, queryable representation linking time-bounded moments (evidence units), entity appearances, relationships, and corpus-level themes. (7:48)

  4. Core Design Principles 9:04

    To build effective video memory, systems must 'ingest once and reason many times,' store primitives (not just answers), ground every claim to a specific timestamp, let intent shape the memory, and keep the layer composable via APIs. (9:04)

Watch on YouTube Full article

Universal 3.5 Pro Demo: Smarter Speech-to-Text with Contextual Awareness thumbnail

· 10:07

Universal 3.5 Pro Demo: Smarter Speech-to-Text with Contextual Awareness

This demo introduces Universal 3.5 Pro, an advanced Speech-to-Text (STT) model designed to significantly boost transcription accuracy through enhanced contextual awareness. Key features include passing domain-specific prompts (e.g., 'cardiology consultation'), applying context to key terms to prevent misapplication, and supporting dynamic mid-call prompt updates via API calls. Furthermore, the model retains conversation history (agent context), allowing it to accurately transcribe user input even in poor audio conditions by understanding the situational flow of a voice agent interaction.

Key takeaways

  1. Contextual Prompting for Domain Accuracy

    Passing detailed information about the audio content (e.g., 'cardiology consultation between Dr. Smith and elderly patient regarding chest pain...') dramatically improves model accuracy within specific domains. The more specific the prompt, the better the results.

  2. Contextual Key Terms 2:00

    Unlike previous methods where key terms were applied blindly, Universal 3.5 Pro allows users to define what a key term represents (e.g., 'The user's name is Zachary Klebanoff'). This prevents the model from incorrectly applying terminology based solely on acoustic similarity.

  3. Dynamic Mid-Call Prompt Updates 2:55

    The prompt can be updated in real time via the API (not available in the playground demo). This is crucial for voice agents, allowing tool calls or external data to adjust the model's context mid-conversation.

  4. Conversation/Agent Context 3:30

    The model retains previous transcriptions and accepts LLM-generated responses from a voice agent as context. This provides situational awareness, improving accuracy even in poor audio conditions and reducing the Word Error Rate (WER) on voice agent datasets.

Watch on YouTube Full article