Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI
This presentation details the architecture of a real-time voice agent harness designed to achieve Frontier-level intelligence while maintaining low latency. The system utilizes a cascaded voice stack, drawing parallels to self-driving car systems, breaking the process into Perception (Transcription), Planning (LLM/Tool Calling), and Control (Speech Synthesis). Key innovations include a streaming speculative transcriber for accuracy, background agents for tool calling, and a prefix cache combined with audio suppression techniques to hide generation latency and ensure seamless, natural conversation flow.
Key takeaways
-
Cascaded Voice Agent Architecture
The system is structured into three layers: Perception (Transcription, converting audio to data), Planning (LLM, processing data and determining actions), and Control (Speech Synthesis, converting text back to natural audio).
-
Streaming Speculative Transcriber
2:32
A hybrid approach combining a fast streaming transcriber (e.g., Flux) with a slower, more accurate batch transcription (e.g., Scribe V2) that uses context to correctly identify entities like names and dates of birth.
-
Background Tool Calling
7:04
To reduce round trips with slow, intelligent LLMs, background agents perform tool calls and inject the results into the main model's context, making the main agent believe it executed the call itself.
-
Prefix Cache for Synthesis
10:57
The prefix cache monitors the model's output stream, checking if audio for a sequence of words already exists from a prior turn. This allows the agent to start speaking immediately from cached audio while the rest of the sentence is being generated.