# Latency Is a Budget. Humanlike Is the Goal. — Jesse Hall, LiveKit

## Executive summary

Building human-like voice agents requires more than just low measured latency; the goal is achieving a natural, human-like conversation flow. The talk argues that latency is a budget, but the focus must be on the entire pipeline's perceived performance, which is affected by compounding errors across the Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS) stack. Key strategies include implementing audio-based turn detection, 'buying time' during tool calls, and aiming for an end-to-end latency of 600 milliseconds or less to feel truly human.

## Key takeaways

- Perceived vs. Measured Latency: Dashboard metrics (like time to first byte or time to first token) are misleading. Callers hear the 'time to first audio' and the overall conversational flow. The goal is to optimize the user experience, not just the component speed.
- The Pipeline Trap: Leaderboards score individual components (STT, LLM, TTS), but real-world agents are pipelines. Errors compound (e.g., an incorrect STT transcription leads the LLM to confidently book the wrong date), making the stack, not the model, the critical focus.
- Buying Time and Turn Detection: To maintain a human feel, agents must 'buy time' (e.g., by stating, 'Let me look up your account') while tools run asynchronously. Furthermore, using audio-based turn detection (which detects changes in pitch/intonation) is crucial for natural interruption handling.
- The 600ms Target: While an end-to-end latency of 1.5 seconds is acceptable, achieving 600 milliseconds is necessary for the agent to feel truly human. The focus should be on the most human-like agent that fits within the latency budget.

## Technical details

- Voice Agent Pipeline Architecture: A voice agent operates as a three-component system: Speech-to-Text (STT) $\rightarrow$ LLM $\rightarrow$ Text-to-Speech (TTS). The overall system performance is defined by the weakest link in this cascade.
- Audio-Based Turn Detection: The speaker introduced a turn detector that analyzes the rhythm, pitch, and intonation of the audio itself, rather than waiting for a full transcript. This method achieved a user cutoff rate of 4.5%, significantly better than the best alternative's 9.9%.
- Benchmarking Methodology: The recommended benchmark tests a complete, working agent against hundreds of simulated callers (interrupting, mumbling, changing minds). Scoring focuses on human-equivalent outcomes (e.g., did the correct reservation land in the database?) rather than just component metrics.
- LiveKit Inference and Model Tuning: The talk showcased a sneak peek of the Gemini 4 31B model on LiveKit Inference. This model was specifically tuned for production voice agents, achieving an estimated 380 milliseconds start time, making it one of the fastest high-quality voice LLMs available.

## Practical implications

- When designing a voice agent, prioritize system-level testing (the full conversation) over component-level testing (individual model scores).
- Architect the system to handle tool calls asynchronously, allowing the agent to 'buy time' by keeping the conversation flowing while background processes run.
- Implement audio-based turn detection to accurately sense when a user is finished speaking, minimizing awkward pauses and cutoffs.
- Set a clear latency budget, aiming for 600ms end-to-end perceived latency, and allocate remaining budget to model quality, not just raw speed.

## Topics

Voice AI, Real-Time Systems, LLM Pipelines, Latency Optimization, Build Engineering, Turn Detection, LiveKit, LiveKit Agents docs, LiveKit Open-Source Benchmark

Source: https://www.youtube.com/watch?v=j4AmIK6OvHc