Latency Is a Budget. Humanlike Is the Goal. — Jesse Hall, LiveKit

Published 2026-10-11 · Duration 17:01

Summary

Building human-like voice agents requires more than just low measured latency; the goal is achieving a natural, human-like conversation flow. The talk argues that latency is a budget, but the focus must be on the entire pipeline's perceived performance, which is affected by compounding errors across the Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS) stack. Key strategies include implementing audio-based turn detection, 'buying time' during tool calls, and aiming for an end-to-end latency of 600 milliseconds or less to feel truly human.

Download summary

Key takeaways

Perceived vs. Measured Latency 0:06
Dashboard metrics (like time to first byte or time to first token) are misleading. Callers hear the 'time to first audio' and the overall conversational flow. The goal is to optimize the user experience, not just the component speed.
The Pipeline Trap 0:04
Leaderboards score individual components (STT, LLM, TTS), but real-world agents are pipelines. Errors compound (e.g., an incorrect STT transcription leads the LLM to confidently book the wrong date), making the stack, not the model, the critical focus.
Buying Time and Turn Detection 0:07
To maintain a human feel, agents must 'buy time' (e.g., by stating, 'Let me look up your account') while tools run asynchronously. Furthermore, using audio-based turn detection (which detects changes in pitch/intonation) is crucial for natural interruption handling.
The 600ms Target 0:11
While an end-to-end latency of 1.5 seconds is acceptable, achieving 600 milliseconds is necessary for the agent to feel truly human. The focus should be on the most human-like agent that fits within the latency budget.

Technical details

Voice Agent Pipeline Architecture 2s
A voice agent operates as a three-component system: Speech-to-Text (STT) $\rightarrow$ LLM $\rightarrow$ Text-to-Speech (TTS). The overall system performance is defined by the weakest link in this cascade.
Audio-Based Turn Detection 10s
The speaker introduced a turn detector that analyzes the rhythm, pitch, and intonation of the audio itself, rather than waiting for a full transcript. This method achieved a user cutoff rate of 4.5%, significantly better than the best alternative's 9.9%.
Benchmarking Methodology 14s
The recommended benchmark tests a complete, working agent against hundreds of simulated callers (interrupting, mumbling, changing minds). Scoring focuses on human-equivalent outcomes (e.g., did the correct reservation land in the database?) rather than just component metrics.
LiveKit Inference and Model Tuning 15s
The talk showcased a sneak peek of the Gemini 4 31B model on LiveKit Inference. This model was specifically tuned for production voice agents, achieving an estimated 380 milliseconds start time, making it one of the fastest high-quality voice LLMs available.

Mentioned resources

LiveKit (Open-Source Real-Time Framework)
LiveKit Agents docs (Documentation)
LiveKit Open-Source Benchmark (Evaluation Tool)

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.