Latency Is a Budget. Humanlike Is the Goal. — Jesse Hall, LiveKit
Building human-like voice agents requires more than just low measured latency; the goal is achieving a natural, human-like conversation flow. The talk argues that latency is a budget, but the focus must be on the entire pipeline's perceived performance, which is affected by compounding errors across the Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS) stack. Key strategies include implementing audio-based turn detection, 'buying time' during tool calls, and aiming for an end-to-end latency of 600 milliseconds or less to feel truly human.
Key takeaways
-
Perceived vs. Measured Latency
0:06
Dashboard metrics (like time to first byte or time to first token) are misleading. Callers hear the 'time to first audio' and the overall conversational flow. The goal is to optimize the user experience, not just the component speed.
-
The Pipeline Trap
0:04
Leaderboards score individual components (STT, LLM, TTS), but real-world agents are pipelines. Errors compound (e.g., an incorrect STT transcription leads the LLM to confidently book the wrong date), making the stack, not the model, the critical focus.
-
Buying Time and Turn Detection
0:07
To maintain a human feel, agents must 'buy time' (e.g., by stating, 'Let me look up your account') while tools run asynchronously. Furthermore, using audio-based turn detection (which detects changes in pitch/intonation) is crucial for natural interruption handling.
-
The 600ms Target
0:11
While an end-to-end latency of 1.5 seconds is acceptable, achieving 600 milliseconds is necessary for the agent to feel truly human. The focus should be on the most human-like agent that fits within the latency budget.