Your Voice Agent is Just a Walkie Talkie — Neil Zeghidour, Gradium
The talk analyzes the evolution of voice agents, arguing that current real-time voice models are fundamentally half-duplex (either listening or speaking). The core technical challenge is achieving full-duplex communication, which involves modeling overlapping speech (like backchanneling). The speaker, Neil Zeghidour, proposes that the most viable path forward is a hybrid architecture: coupling a small, highly natural, full-duplex speech-to-speech (S2S) interface with a powerful, asynchronous background text LLM to handle all complex reasoning and tool calling. This approach mitigates the inherent trade-off where improving naturalness sacrifices intelligence.
Key takeaways
-
Evolution of Voice Agents
0:10
Voice agent technology has progressed through constrained, closed-ended dialogue (Siri, 2011) to open-ended conversational models (OpenAI Voice Mode), and finally to agentic systems capable of real actions (e.g., ordering food).
-
The Full-Duplex Challenge
12:28
Human conversation is full-duplex, allowing for overlapping speech and backchanneling (e.g., 'Mhm, yeah'). Current S2S models, even with low latency, are limited by fundamental turn-taking mechanisms, making them feel unnatural.
-
The Intelligence vs. Naturalness Trade-off
18:20
There is a fundamental tension: every gain in naturalness (e.g., moving from cascaded STT/LLM/TTS to S2S) requires dedicating model capacity (weights) to audio modalities, which reduces the model's overall intelligence and reasoning capability.
-
The Hybrid Solution
18:40
The recommended approach is to split the system: use a small, on-device, full-duplex S2S model for natural conversation flow, while delegating all complex reasoning, tool calling, and agentic capabilities to a separate, powerful background text LLM.