# Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI

## Executive summary

This presentation details the architecture of a real-time voice agent harness designed to achieve Frontier-level intelligence while maintaining low latency. The system utilizes a cascaded voice stack, drawing parallels to self-driving car systems, breaking the process into Perception (Transcription), Planning (LLM/Tool Calling), and Control (Speech Synthesis). Key innovations include a streaming speculative transcriber for accuracy, background agents for tool calling, and a prefix cache combined with audio suppression techniques to hide generation latency and ensure seamless, natural conversation flow.

## Key takeaways

- Cascaded Voice Agent Architecture: The system is structured into three layers: Perception (Transcription, converting audio to data), Planning (LLM, processing data and determining actions), and Control (Speech Synthesis, converting text back to natural audio).
- Streaming Speculative Transcriber: A hybrid approach combining a fast streaming transcriber (e.g., Flux) with a slower, more accurate batch transcription (e.g., Scribe V2) that uses context to correctly identify entities like names and dates of birth.
- Background Tool Calling: To reduce round trips with slow, intelligent LLMs, background agents perform tool calls and inject the results into the main model's context, making the main agent believe it executed the call itself.
- Prefix Cache for Synthesis: The prefix cache monitors the model's output stream, checking if audio for a sequence of words already exists from a prior turn. This allows the agent to start speaking immediately from cached audio while the rest of the sentence is being generated.

## Technical details

- Perception Layer (Transcription): Uses a streaming speculative transcriber, layering a fast engine over an accurate, context-aware batch transcriber (Scribe V2). The slower engine improves accuracy by understanding the context of the query (e.g., distinguishing a name from a date of birth).
- Planning Layer (LLM/Tool Calling): Background agents handle tool calling asynchronously. The results are pushed back into the main model's context, allowing the main agent to proceed as if it initiated the tool call, thereby minimizing perceived latency.
- Control Layer (Speech Synthesis): Achieves seamless audio output using a prefix cache and a standard TTS provider (e.g., Cartesia). The cache emits pre-generated audio for the initial words, while the provider generates the remainder. The system then suppresses the overlapping audio from the provider to ensure a seamless join without a detectable seam.

## Practical implications

- Enables real-time voice agents with high intelligence by optimizing latency at every stage of the voice stack.
- The architecture allows for the decoupling and independent optimization of transcription, planning, and synthesis layers.
- The use of background processing and caching techniques makes the agent's response feel instantaneous and natural to the user.

## Topics

Voice Agents, Natural Language Processing, Speech Synthesis, LLM Architecture, Real-Time Systems, EliseAI, Xnor.ai

Source: https://www.youtube.com/watch?v=MBHOH1NmDqc