How to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit
Summary
Building human-like voice agents, especially for sensitive fields like financial services, requires moving beyond simple component performance metrics (like model leaderboards). The focus must shift to the entire conversational flow, latency management, and the ability to handle real-world interruptions. Key strategies include measuring time to first audio/sentence, utilizing asynchronous tool calls to maintain conversation flow during slow back-end operations, and implementing advanced turn detection that analyzes audio characteristics, not just transcripts.
Key takeaways
-
Measure Conversational Flow, Not Components
Leaderboards score individual components (e.g., TTS, LLM), but real-time agents are a system. The goal is to score the entire conversation, not just the individual models.
-
Prioritize Time to First Audio/Sentence
10:46
For optimal user experience, measure 'time to first audio' (the time the user actually hears sound) rather than 'time to first byte' (a text-based metric). Similarly, measure 'time to first sentence' for LLMs, as TTS models require a full sentence to begin speaking.
-
Handle Latency with Asynchronous Calls
16:54
When back-end tasks (like looking up a transaction) take significant time, agents must not go silent. Implementing asynchronous tool calls (using `awaiting` a tool call) allows the agent to keep the conversation flowing, mimicking human multitasking.
-
Use Audio-Aware Turn Detection
20:33
Effective turn detection must listen to the audio—analyzing rhythm, pitch, and intonation—and not just the transcript. This prevents the agent from prematurely cutting off the user, which is crucial for natural interaction.
-
Benchmark the Full Stack with Simulated Callers
The only reliable way to select the best model stack is to build a personalized benchmark. This involves running simulated callers (who can interrupt, change their mind, or mumble) against the complete agent, testing the entire pipeline (audio-based and text-based).
Technical details
-
Voice Agent Architectures
346s
Traditional voice agents use a cascaded pipeline (Speech-to-Text -> LLM -> Text-to-Speech). Newer, more advanced systems are moving toward real-time speech-to-speech models, though the traditional pipeline remains common.
-
Latency Budget
The ideal latency budget is critical for human-like interaction. While a 1.5-second gap is acceptable, achieving 500-600ms between the caller's last syllable and the agent's first audible response feels truly natural.
-
Asynchronous Tool Calling
1014s
To prevent the conversation from stalling during slow back-end operations, the agent must use asynchronous processing. The ability to 'await' a tool call allows the conversation thread to continue while the tool executes in the background.
-
Evaluation Methodology
Benchmarking must test the final state (e.g., did the correct reservation hit the database?) and the process (e.g., did the agent take the minimum number of steps?) rather than just completion status. This prevents agents from 'gaming' the outcome.
Mentioned resources
- LiveKit
- AIE CODE summit
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.