The Video Signal

Topic

Speech Synthesis

All digests tagged Speech Synthesis

Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI thumbnail

· 13:14

Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI

This presentation details the architecture of a real-time voice agent harness designed to achieve Frontier-level intelligence while maintaining low latency. The system utilizes a cascaded voice stack, drawing parallels to self-driving car systems, breaking the process into Perception (Transcription), Planning (LLM/Tool Calling), and Control (Speech Synthesis). Key innovations include a streaming speculative transcriber for accuracy, background agents for tool calling, and a prefix cache combined with audio suppression techniques to hide generation latency and ensure seamless, natural conversation flow.

Key takeaways

  1. Cascaded Voice Agent Architecture

    The system is structured into three layers: Perception (Transcription, converting audio to data), Planning (LLM, processing data and determining actions), and Control (Speech Synthesis, converting text back to natural audio).

  2. Streaming Speculative Transcriber 2:32

    A hybrid approach combining a fast streaming transcriber (e.g., Flux) with a slower, more accurate batch transcription (e.g., Scribe V2) that uses context to correctly identify entities like names and dates of birth.

  3. Background Tool Calling 7:04

    To reduce round trips with slow, intelligent LLMs, background agents perform tool calls and inject the results into the main model's context, making the main agent believe it executed the call itself.

  4. Prefix Cache for Synthesis 10:57

    The prefix cache monitors the model's output stream, checking if audio for a sequence of words already exists from a prior turn. This allows the agent to start speaking immediately from cached audio while the rest of the sentence is being generated.

Watch on YouTube Full article

5 Voice Agent Failure Modes You'll Hit in Week One — Venky B, Plivo thumbnail

· 26:46

5 Voice Agent Failure Modes You'll Hit in Week One — Venky B, Plivo

This talk details five critical failure modes encountered when deploying voice AI agents from a proof-of-concept (PoC) environment to production scale. The core message emphasizes that successful production deployment requires shifting focus from end-to-end transcript processing to structured data collection, rigorous latency management, and building robust normalization layers between the LLM and Text-to-Speech (TTS) components. Key technical recommendations include using smaller, self-hosted open-source models for low latency, treating data collection as unit-testable field validation, and normalizing all output before synthesis.

Key takeaways

  1. Latency and Time to First Audio (TTFA) 9:54

    Industry average TTFA often falls between 750ms and 1,200ms, leading to user drop-off. To improve this, the speaker recommends using smaller, self-hosted open-source models (e.g., Quen 3.5, Gemma 4) to target under 300ms, balancing cost, intelligence, and latency.

  2. Data Collection Must Be Structured 19:20

    Instead of treating input as a single transcript, data collection should be modeled like structured fields (e.g., using Pydantic or Zod). This approach significantly increases accuracy (from ~30% to mid-90s) by allowing validation and error handling per field (e.g., phone numbers, addresses).

  3. Normalization Layer is Critical 24:30

    Never feed raw LLM output directly to a TTS engine. A dedicated normalization layer must be implemented to strip markdown/emojis, apply custom pronunciation dictionaries, and control speaking speed (e.g., 0.7x or 0.8x) to ensure proper pronunciation of entities.

  4. Transcription is Brittle by Default 20:40

    Transcription engines struggle with proper nouns, jargon, and code-switched languages. Solutions include implementing dynamic keyword boosting and post-processing the raw transcript using an LLM layer, which provides necessary domain context.

Watch on YouTube Full article