# How to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit

## Executive summary

Building human-like voice agents, especially for sensitive fields like financial services, requires moving beyond simple component performance metrics (like model leaderboards). The focus must shift to the entire conversational flow, latency management, and the ability to handle real-world interruptions. Key strategies include measuring time to first audio/sentence, utilizing asynchronous tool calls to maintain conversation flow during slow back-end operations, and implementing advanced turn detection that analyzes audio characteristics, not just transcripts.

## Key takeaways

- Measure Conversational Flow, Not Components: Leaderboards score individual components (e.g., TTS, LLM), but real-time agents are a system. The goal is to score the entire conversation, not just the individual models.
- Prioritize Time to First Audio/Sentence: For optimal user experience, measure 'time to first audio' (the time the user actually hears sound) rather than 'time to first byte' (a text-based metric). Similarly, measure 'time to first sentence' for LLMs, as TTS models require a full sentence to begin speaking.
- Handle Latency with Asynchronous Calls: When back-end tasks (like looking up a transaction) take significant time, agents must not go silent. Implementing asynchronous tool calls (using `awaiting` a tool call) allows the agent to keep the conversation flowing, mimicking human multitasking.
- Use Audio-Aware Turn Detection: Effective turn detection must listen to the audio—analyzing rhythm, pitch, and intonation—and not just the transcript. This prevents the agent from prematurely cutting off the user, which is crucial for natural interaction.
- Benchmark the Full Stack with Simulated Callers: The only reliable way to select the best model stack is to build a personalized benchmark. This involves running simulated callers (who can interrupt, change their mind, or mumble) against the complete agent, testing the entire pipeline (audio-based and text-based).

## Technical details

- Voice Agent Architectures: Traditional voice agents use a cascaded pipeline (Speech-to-Text -> LLM -> Text-to-Speech). Newer, more advanced systems are moving toward real-time speech-to-speech models, though the traditional pipeline remains common.
- Latency Budget: The ideal latency budget is critical for human-like interaction. While a 1.5-second gap is acceptable, achieving 500-600ms between the caller's last syllable and the agent's first audible response feels truly natural.
- Asynchronous Tool Calling: To prevent the conversation from stalling during slow back-end operations, the agent must use asynchronous processing. The ability to 'await' a tool call allows the conversation thread to continue while the tool executes in the background.
- Evaluation Methodology: Benchmarking must test the final state (e.g., did the correct reservation hit the database?) and the process (e.g., did the agent take the minimum number of steps?) rather than just completion status. This prevents agents from 'gaming' the outcome.

## Practical implications

- When designing a voice agent, treat it as a real-time system, not just a chatbot with audio output.
- Implement rigorous testing that simulates real-world noise, interruptions, and goal changes.
- Focus on optimizing the entire pipeline (STT, LLM, TTS, Tool Calls) to fit within the 1.5-second latency budget.
- Use open-source simulation tools to build a personalized benchmark for your specific use case, rather than relying on general leaderboards.

## Topics

Voice Agents, Real-Time AI, Latency Optimization, Conversational AI, System Benchmarking, LiveKit, AIE CODE summit

Source: https://www.youtube.com/watch?v=OmyqelTwpOo