The Video Signal

Topic

Voice AI

All digests tagged Voice AI

100x More Reliable Voice Agents thumbnail

· 14:35

100x More Reliable Voice Agents

Unmute is introduced as an open, declarative standard for building voice agents, designed to solve the critical problem of vendor lock-in and architectural fragility in the current voice AI landscape. Current agents are often embedded within specific platforms (e.g., ElevenLabs, VAPI, LiveKit), making them non-portable. Unmute shifts the paradigm by declaring the agent's behavior, tool calls, data flows, and failure handling in a single, compiled package, making the LLM an optional resource rather than the sole decision-making spine. This approach ensures structural guarantees and portability across different cloud runtimes.

Key takeaways

  1. Declarative Execution and Portability 2:00

    Unmute allows the agent's behavior to be defined in a single, portable package that compiles to various runtimes (e.g., PipeChat Cloud, LiveKit Cloud, or Slang's architecture). This means the agent's core logic remains version-controlled IP, regardless of the underlying infrastructure or cloud region.

  2. Addressing LLM Over-Reliance 3:00

    Traditional agents assume the LLM is the decision-making spine, leading to unpredictable, expensive, and difficult-to-govern behavior. Unmute enforces execution through declared steps, making the LLM an optional resource that must 'earn its place' in the call.

  3. Structural Guarantees over Instructions 4:00

    The standard moves beyond the false choice between rigid IVR and complex single-prompt agents. By splitting the work into discrete, context-limited steps, the system achieves determinism—allowing specific rules (like a regulated disclosure) to be locked down while other parts remain generative, all within the same call.

Watch on YouTube Full article

🪄 Gemini Live API in action thumbnail

· 1:32

🪄 Gemini Live API in action

This video demonstrates the new capabilities of the Gemini Live API, focusing on advanced features designed for real-time, context-aware interactions. Key additions include async function calling for faster execution, Proactive Audio for relevant speaking, and the ability to inject context using `sendClientContent`. The API also showcases frontier-level background reasoning, which was demonstrated by switching to a 'Max' high reasoning model for improved creative output.

Key takeaways

  1. Async Function Calling

    Introduced for faster and more efficient execution of tasks within the Live API.

  2. Proactive Audio

    Ensures the agent only speaks when relevant to the conversation, improving the user experience.

  3. Context Injection

    The ability to inject context using `sendClientContent` allows the agent to maintain relevance and focus during long conversations.

  4. Enhanced Reasoning

    Demonstrated by switching to a 'Max' high reasoning model, significantly improving the quality and detail of creative outputs (e.g., SVG generation).

Watch on YouTube Full article

Act, Confirm, or Stop? Smarter behavior for AI assistants, wearables & robots — Amit Desai, Roku thumbnail

· 20:25

Act, Confirm, or Stop? Smarter behavior for AI assistants, wearables & robots — Amit Desai, Roku

The presentation argues that improving voice AI user experience requires focusing on a second, often neglected dimension: system behavior under uncertainty. While increasing accuracy (Knob One) is critical, the system's ability to intelligently decide what to do when it is unsure (Knob Two) can yield greater user satisfaction. This is quantified using the Outcome User Cost Heuristic (OUCH), which minimizes the total user effort by assigning differential costs to various bad outcomes (e.g., playing the wrong song vs. simply stating 'I did not understand').

Key takeaways

  1. The Two Knobs of Voice AI Improvement 0:03

    User satisfaction can be improved by increasing technical accuracy (Knob One) or by optimizing the system's decision-making process when confidence is low (Knob Two). The latter is often overlooked.

  2. The Outcome User Cost Heuristic (OUCH) 0:10

    Instead of treating all errors equally, OUCH minimizes the total user cost by quantifying the relative pain of different bad outcomes (e.g., the effort required to stop a wrong song vs. the time taken to hear 'Sorry, I did not understand').

  3. Adding Conversational Behavior 0:13

    Introducing a third behavior—confirming the guess out loud (e.g., 'Did you mean ABC?')—splits the confidence range into three regions (Stop, Confirm, Act) and further lowers the overall user cost.

Watch on YouTube Full article

"My name is... my name is...": A Linguistic Map for Voice Agents — Midam Kim, ServiceNow thumbnail

· 15:05

"My name is... my name is...": A Linguistic Map for Voice Agents — Midam Kim, ServiceNow

Midam Kim presents a linguistic framework for diagnosing failures in voice AI, arguing that these failures are not isolated bugs but structured issues. She proposes that human communication is a 'joint activity' involving the continuous updating of a 'mental model.' The framework maps this process onto two channels (listening and speaking) and four interdependent levels: sounds, words, interaction, and mental model. Successful voice AI requires holistic orchestration across all these layers, rather than optimizing components (like ASR or TTS) in isolation.

Key takeaways

  1. Voice AI is a Joint Activity 5:00

    Human communication is a joint activity where both parties contribute sounds and words, continuously updating a shared mental model. Voice AI systems must replicate this joint nature to be effective.

  2. The Linguistic Framework 11:54

    The system must be analyzed across two channels (listening/speaking) and four interdependent levels: sounds, words, interaction, and mental model. Failure in one area (e.g., STT failure at the sound level) impacts the entire system.

  3. Mental Model Accumulation

    Unlike text chat where history remains visible, in voice interactions, sounds and words vanish. The only persistent element that matters for user satisfaction is the user's accumulating mental model.

  4. System Adaptability is Key

    The system must be designed to be dynamic, adapting to context, emotion, and language change over the course of the call, rather than functioning as a static pipeline.

Watch on YouTube Full article

5 Voice Agent Failure Modes You'll Hit in Week One — Venky B, Plivo thumbnail

· 26:46

5 Voice Agent Failure Modes You'll Hit in Week One — Venky B, Plivo

This talk details five critical failure modes encountered when deploying voice AI agents from a proof-of-concept (PoC) environment to production scale. The core message emphasizes that successful production deployment requires shifting focus from end-to-end transcript processing to structured data collection, rigorous latency management, and building robust normalization layers between the LLM and Text-to-Speech (TTS) components. Key technical recommendations include using smaller, self-hosted open-source models for low latency, treating data collection as unit-testable field validation, and normalizing all output before synthesis.

Key takeaways

  1. Latency and Time to First Audio (TTFA) 9:54

    Industry average TTFA often falls between 750ms and 1,200ms, leading to user drop-off. To improve this, the speaker recommends using smaller, self-hosted open-source models (e.g., Quen 3.5, Gemma 4) to target under 300ms, balancing cost, intelligence, and latency.

  2. Data Collection Must Be Structured 19:20

    Instead of treating input as a single transcript, data collection should be modeled like structured fields (e.g., using Pydantic or Zod). This approach significantly increases accuracy (from ~30% to mid-90s) by allowing validation and error handling per field (e.g., phone numbers, addresses).

  3. Normalization Layer is Critical 24:30

    Never feed raw LLM output directly to a TTS engine. A dedicated normalization layer must be implemented to strip markdown/emojis, apply custom pronunciation dictionaries, and control speaking speed (e.g., 0.7x or 0.8x) to ensure proper pronunciation of entities.

  4. Transcription is Brittle by Default 20:40

    Transcription engines struggle with proper nouns, jargon, and code-switched languages. Solutions include implementing dynamic keyword boosting and post-processing the raw transcript using an LLM layer, which provides necessary domain context.

Watch on YouTube Full article

Tolan: Voice-First AI Companion — Paula Dozsa, Tolan thumbnail

· 15:08

Tolan: Voice-First AI Companion — Paula Dozsa, Tolan

Paula Dozsa, an engineer on the Tolan team, details the unique engineering challenges of building a voice-first AI companion compared to traditional text-based LLM applications. She emphasizes that voice introduces 'conversational volatility' (fast turns, interruptions) which requires fundamental shifts in pipeline design, including smart turn detection, tiered model routing based on emotional stakes, and rebuilding context per turn rather than relying on a continuous cache. The talk also covers using advanced AI agents (like Claude) to build the product itself, achieving significant improvements in stability and feature development.

Key takeaways

  1. Voice vs. Text LLM Assumptions 4:05

    Text chat assumes slow turns and stable context, while voice is characterized by fast turns and volatile context (interruptions, subject changes mid-sentence). This volatility requires building for messy, real-world speech patterns.

  2. Optimizing for Interruptions 5:38

    Instead of minimizing interruptions, the team focused on building smart turn-taking that reads speech patterns to prevent early, incorrect agent interventions. They paid an extra 60 milliseconds of latency to achieve this.

  3. Memory as Retrieval System 12:32

    To handle volatile context, memory is treated as a retrieval system, not a transcript. Facts and preferences are embedded, stored in a vector database (with sub-50ms lookups), and compressed nightly to resolve contradictions and merge duplicates.

  4. Context Reassembly

    Context must be reassembled from parts every single turn (summary, user persona, retrieved memories, tone guidance) because reusing old context is a 'trap' when the user pivots subjects.

Watch on YouTube Full article

Your Voice Agent is Just a Walkie Talkie — Neil Zeghidour, Gradium thumbnail

· 19:14

Your Voice Agent is Just a Walkie Talkie — Neil Zeghidour, Gradium

The talk analyzes the evolution of voice agents, arguing that current real-time voice models are fundamentally half-duplex (either listening or speaking). The core technical challenge is achieving full-duplex communication, which involves modeling overlapping speech (like backchanneling). The speaker, Neil Zeghidour, proposes that the most viable path forward is a hybrid architecture: coupling a small, highly natural, full-duplex speech-to-speech (S2S) interface with a powerful, asynchronous background text LLM to handle all complex reasoning and tool calling. This approach mitigates the inherent trade-off where improving naturalness sacrifices intelligence.

Key takeaways

  1. Evolution of Voice Agents 0:10

    Voice agent technology has progressed through constrained, closed-ended dialogue (Siri, 2011) to open-ended conversational models (OpenAI Voice Mode), and finally to agentic systems capable of real actions (e.g., ordering food).

  2. The Full-Duplex Challenge 12:28

    Human conversation is full-duplex, allowing for overlapping speech and backchanneling (e.g., 'Mhm, yeah'). Current S2S models, even with low latency, are limited by fundamental turn-taking mechanisms, making them feel unnatural.

  3. The Intelligence vs. Naturalness Trade-off 18:20

    There is a fundamental tension: every gain in naturalness (e.g., moving from cascaded STT/LLM/TTS to S2S) requires dedicating model capacity (weights) to audio modalities, which reduces the model's overall intelligence and reasoning capability.

  4. The Hybrid Solution 18:40

    The recommended approach is to split the system: use a small, on-device, full-duplex S2S model for natural conversation flow, while delegating all complex reasoning, tool calling, and agentic capabilities to a separate, powerful background text LLM.

Watch on YouTube Full article

Event Recap: Build Smarter Voice Agents - New York Edition thumbnail

· 29:13

Event Recap: Build Smarter Voice Agents - New York Edition

This recap details the complexities of building and deploying production-grade voice AI agents across two distinct sectors: professional networking (Boardy) and regulated healthcare (Flagler Health). Key challenges discussed include maintaining conversational flow, establishing user trust, managing multi-party video meeting interactions, and ensuring subsecond latency for natural conversation. The discussion highlights the difference between highly structured, goal-oriented flows (healthcare) and highly conversational, relationship-driven interactions (networking).

Key takeaways

  1. Design Flow Differences 10:20

    Healthcare voice agents require highly structured, step-by-step flows with strict guardrails (e.g., collecting insurance info) to prevent medical advice or deviation. Conversely, networking agents are designed to handle highly conversational, open-ended interactions to facilitate connections.

  2. The Importance of Trust and Disclosure 21:20

    Building user trust is critical. Speakers emphasized that being upfront and immediately disclosing that the user is speaking to an AI (e.g., 'I'm Sarah and AI') is essential to prevent user frustration and loss of trust.

  3. Technical Challenge: Multi-Party Meetings 24:10

    Handling voice agents in multi-person video meetings (like Google Meet) is technically difficult. The primary challenge is determining when the agent should speak (turn-taking) to avoid false positives (randomly jumping in) or false negatives (failing to reply).

  4. Achieving Low Latency 25:00

    To feel like a natural conversation, the system must achieve subsecond latency. This requires advanced architecture, such as preemptively generating the entire voice pipeline while the user is speaking.

Watch on YouTube Full article

Build voice-first apps with Gemini 3.5 Transcribe thumbnail

· 1:44

Build voice-first apps with Gemini 3.5 Transcribe

Google launched Gemini 3.5 Transcribe, an advanced LLM-based model designed for building voice-first applications. This model is available via both the Interactions API and the Live API, offering fast, contextually accurate transcription of multi-speaker recordings. Key strengths include superior recognition of structured data like email addresses and phone numbers, as well as robust support for over 70 different languages.

Key takeaways

  1. Model Availability

    Gemini 3.5 Transcribe is available on both the Interactions API and the Live API.

  2. Structured Data Recognition

    The LLM-based model excels at transcribing alphanumerics, such as email addresses (e.g., thorwebdev@google.com) and recognizing correct US phone number formats.

  3. Multi-Language Support

    The model can recognize and transcribe over 70 different languages, even when language hints are set to English.

Watch on YouTube Full article

⏭️ Forward Deployed: Voice AI on what works in 2026 thumbnail

· 36:31

⏭️ Forward Deployed: Voice AI on what works in 2026

The discussion explores the advanced engineering challenges of deploying real-time voice AI agents in enterprise settings. Speakers emphasize that achieving human-like performance requires complex, multi-stage cascaded pipelines (Speech-to-Text $\rightarrow$ LLM $\rightarrow$ Text-to-Speech), rather than simple end-to-end Voice-to-Voice models. Key challenges include managing latency trade-offs, ensuring reliability through model fallbacks, solving non-trivial problems like turn-taking, and architecting systems to handle complex context while maintaining low cost.

Key takeaways

  1. Architectural Standard: Cascaded Pipeline 3:45

    The current state-of-the-art for reliable voice agents is a cascaded pipeline (Speech-to-Text $\rightarrow$ LLM $\rightarrow$ Text-to-Speech). While Voice-to-Voice models are appealing, they currently lack the necessary reliability and control required for enterprise use cases. [0:03:45]

  2. The Latency vs. Intelligence Trade-off 3:15

    There is an inherent trade-off between the intelligence of the response and the latency. Achieving highly intelligent responses often results in slower processing, which must be managed to feel natural to the end user. [0:03:15]

  3. Complexity of Conversation Flow 3:05

    Solving conversational nuances like turn-taking (knowing when a pause means thinking versus ending the conversation) is not trivial. Furthermore, systems must account for different use cases: outbound calls are easier because guardrails can be strict, while inbound calls require handling unknown user context. [0:03:25]

  4. Optimizing LLM Context and Cost 3:58

    Due to the limitations of LLMs (forgetting information in the middle of a massive prompt), advanced architectures must employ context optimization techniques. This includes using specialized models or implementing 'compaction' methods, which is becoming critical for both performance and unit economics. [0:03:58]

Watch on YouTube Full article

2nd Place Winner: Coding Agent Calls Developer to Pitch Launch Strategy thumbnail

· 5:12

2nd Place Winner: Coding Agent Calls Developer to Pitch Launch Strategy

The video demonstrates an autonomous AI agent designed for product positioning strategy that operates while the developer is away (AFK). The agent handles routine tasks but utilizes a defined escalation matrix to call the human developer only when faced with non-reversible, high-stakes decisions. This process not only facilitates real-time discussion via voice call but also ensures all resulting decisions and follow-up action items are automatically logged back into the project documentation for transparency.

Key takeaways

  1. Autonomous AFK Operation

    The agent is instructed to run autonomously, completing all tasks it can handle without human intervention. It also checks working hours to prevent calling outside designated times.

  2. Strategic Escalation Matrix 1:40

    When the agent reaches a critical decision point (e.g., Lead on Value vs. Lead on Price), it triggers an escalation, presenting structured options and recommendations rather than asking for generic input.

  3. Decision Logging and Transparency

    Following the human decision (e.g., 'Lead on Value'), the agent automatically logs the approved decision and creates a follow-up task (e.g., 'follow up in 7 days') directly into the project files, ensuring decisions are never lost within transcripts.

Watch on YouTube Full article

1st Place Winner: Coding Agent Calls Developer to Resolve Code Block thumbnail

· 6:17

1st Place Winner: Coding Agent Calls Developer to Resolve Code Block

The demo showcases an advanced AI coding agent that autonomously handles a critical bug fix in a checkout API. When faced with a technical decision requiring human judgment—specifically, whether to maintain backward compatibility (Option A) or implement a clean refactor causing breaking changes (Option B)—the agent initiates an automated phone call to the developer for real-time guidance and execution.

Key takeaways

  1. Autonomous Agent Setup

    The setup involves running a coding agent via the Claude Code CLI, monitored by the Vocal Bridge dashboard, targeting a validation bug across five checkout API handlers (e.g., create order, apply coupon).

  2. Decision Point Triggered 3:26

    The agent identifies that fixing the bug requires a judgment call: Option A maintains backward compatibility but involves code duplication; Option B is a clean refactor but introduces a breaking change to the error format.

  3. Human-in-the-Loop Communication 1:52

    Instead of guessing, the agent initiates an outbound phone call (via VocalBridgeAI) to present the technical trade-offs and obtain a decision from the developer while they are away from their keyboard.

  4. Automated Execution

    Upon receiving the final verbal confirmation (Option B), the agent automatically executes the chosen path, logs the decision, and updates the code base without manual developer intervention.

Watch on YouTube Full article

Build Smarter Voice Agents thumbnail

· 46:14

Build Smarter Voice Agents

This panel discussion provides deep insights into building production-grade voice AI agents. Key architectural recommendations favor the deconstructed cascading pipeline (ASR $\rightarrow$ LLM $\rightarrow$ TTS) due to its superior flexibility for optimization and model swapping. Engineers must prioritize managing latency within a 1–1.5 second budget, implementing robust fallback systems across all stack components (ASR, LLM, TTS), and utilizing advanced context management techniques like 'Scratchpads' to maintain conversational continuity over long interactions.

Key takeaways

  1. Architectural Choice: Cascading Pipeline 1:45

    The cascading architecture is preferred because it allows for individual optimization of the ASR, LLM, and TTS layers. This modularity provides greater flexibility than a full Speech-to-Speech (S2S) stack when integrating new models or optimizing specific components.

  2. Latency Management 4:23

    The 'golden metric' for voice agent response time is between 1 to 1.5 seconds. Exceeding this budget can be unnerving for users, making latency a primary design constraint over pure accuracy in many cases.

  3. Context and Memory Management 18:05

    To prevent negative user sentiment from repeating information, agents must implement context stores (e.g., 'Scratchpads') to track customer profiles, preferences, and key facts across multiple turns or sessions.

Watch on YouTube Full article

Agent Context Carryover in Livekit Tutorial thumbnail

· 8:07

Agent Context Carryover in Livekit Tutorial

This tutorial demonstrates 'Agent Context Carryover,' a feature released for LiveKit using AssemblyAI's Universal 3.5 Pro model. This capability automatically feeds conversational context into the speech-to-text model, significantly boosting transcription accuracy—especially for proper nouns and key terms—without requiring manual plumbing or explicit context prompting from the developer.

Key takeaways

  1. Agent Context Carryover Functionality

    The feature automatically provides conversational context to the model, improving transcription accuracy when building voice agents on LiveKit. This is achieved by enabling a single parameter rather than implementing complex key term handling.

  2. Implementation Simplicity

    AssemblyAI's LiveKit plugin handles the necessary context plumbing automatically, allowing developers to gain accuracy benefits without setting up custom logic for key terms or sending agent messages manually.

Watch on YouTube Full article

Universal 3.5 Pro Demo: Smarter Speech-to-Text with Contextual Awareness thumbnail

· 10:07

Universal 3.5 Pro Demo: Smarter Speech-to-Text with Contextual Awareness

This demo introduces Universal 3.5 Pro, an advanced Speech-to-Text (STT) model designed to significantly boost transcription accuracy through enhanced contextual awareness. Key features include passing domain-specific prompts (e.g., 'cardiology consultation'), applying context to key terms to prevent misapplication, and supporting dynamic mid-call prompt updates via API calls. Furthermore, the model retains conversation history (agent context), allowing it to accurately transcribe user input even in poor audio conditions by understanding the situational flow of a voice agent interaction.

Key takeaways

  1. Contextual Prompting for Domain Accuracy

    Passing detailed information about the audio content (e.g., 'cardiology consultation between Dr. Smith and elderly patient regarding chest pain...') dramatically improves model accuracy within specific domains. The more specific the prompt, the better the results.

  2. Contextual Key Terms 2:00

    Unlike previous methods where key terms were applied blindly, Universal 3.5 Pro allows users to define what a key term represents (e.g., 'The user's name is Zachary Klebanoff'). This prevents the model from incorrectly applying terminology based solely on acoustic similarity.

  3. Dynamic Mid-Call Prompt Updates 2:55

    The prompt can be updated in real time via the API (not available in the playground demo). This is crucial for voice agents, allowing tool calls or external data to adjust the model's context mid-conversation.

  4. Conversation/Agent Context 3:30

    The model retains previous transcriptions and accepts LLM-generated responses from a voice agent as context. This provides situational awareness, improving accuracy even in poor audio conditions and reducing the Word Error Rate (WER) on voice agent datasets.

Watch on YouTube Full article