Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI
Summary
This presentation details the architecture of a real-time voice agent harness designed to achieve Frontier-level intelligence while maintaining low latency. The system utilizes a cascaded voice stack, drawing parallels to self-driving car systems, breaking the process into Perception (Transcription), Planning (LLM/Tool Calling), and Control (Speech Synthesis). Key innovations include a streaming speculative transcriber for accuracy, background agents for tool calling, and a prefix cache combined with audio suppression techniques to hide generation latency and ensure seamless, natural conversation flow.
Key takeaways
-
Cascaded Voice Agent Architecture
The system is structured into three layers: Perception (Transcription, converting audio to data), Planning (LLM, processing data and determining actions), and Control (Speech Synthesis, converting text back to natural audio).
-
Streaming Speculative Transcriber
2:32
A hybrid approach combining a fast streaming transcriber (e.g., Flux) with a slower, more accurate batch transcription (e.g., Scribe V2) that uses context to correctly identify entities like names and dates of birth.
-
Background Tool Calling
7:04
To reduce round trips with slow, intelligent LLMs, background agents perform tool calls and inject the results into the main model's context, making the main agent believe it executed the call itself.
-
Prefix Cache for Synthesis
10:57
The prefix cache monitors the model's output stream, checking if audio for a sequence of words already exists from a prior turn. This allows the agent to start speaking immediately from cached audio while the rest of the sentence is being generated.
Technical details
-
Perception Layer (Transcription)
152s
Uses a streaming speculative transcriber, layering a fast engine over an accurate, context-aware batch transcriber (Scribe V2). The slower engine improves accuracy by understanding the context of the query (e.g., distinguishing a name from a date of birth).
-
Planning Layer (LLM/Tool Calling)
424s
Background agents handle tool calling asynchronously. The results are pushed back into the main model's context, allowing the main agent to proceed as if it initiated the tool call, thereby minimizing perceived latency.
-
Control Layer (Speech Synthesis)
606s
Achieves seamless audio output using a prefix cache and a standard TTS provider (e.g., Cartesia). The cache emits pre-generated audio for the initial words, while the provider generates the remainder. The system then suppresses the overlapping audio from the provider to ensure a seamless join without a detectable seam.
Mentioned resources
- EliseAI
- Xnor.ai
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.