# Your Voice Agent is Just a Walkie Talkie — Neil Zeghidour, Gradium

## Executive summary

The talk analyzes the evolution of voice agents, arguing that current real-time voice models are fundamentally half-duplex (either listening or speaking). The core technical challenge is achieving full-duplex communication, which involves modeling overlapping speech (like backchanneling). The speaker, Neil Zeghidour, proposes that the most viable path forward is a hybrid architecture: coupling a small, highly natural, full-duplex speech-to-speech (S2S) interface with a powerful, asynchronous background text LLM to handle all complex reasoning and tool calling. This approach mitigates the inherent trade-off where improving naturalness sacrifices intelligence.

## Key takeaways

- Evolution of Voice Agents: Voice agent technology has progressed through constrained, closed-ended dialogue (Siri, 2011) to open-ended conversational models (OpenAI Voice Mode), and finally to agentic systems capable of real actions (e.g., ordering food).
- The Full-Duplex Challenge: Human conversation is full-duplex, allowing for overlapping speech and backchanneling (e.g., 'Mhm, yeah'). Current S2S models, even with low latency, are limited by fundamental turn-taking mechanisms, making them feel unnatural.
- The Intelligence vs. Naturalness Trade-off: There is a fundamental tension: every gain in naturalness (e.g., moving from cascaded STT/LLM/TTS to S2S) requires dedicating model capacity (weights) to audio modalities, which reduces the model's overall intelligence and reasoning capability.
- The Hybrid Solution: The recommended approach is to split the system: use a small, on-device, full-duplex S2S model for natural conversation flow, while delegating all complex reasoning, tool calling, and agentic capabilities to a separate, powerful background text LLM.

## Technical details

- Audio Tokenization and Compression: Training LLMs on raw audio (a waveform) is computationally prohibitive. For example, 8 words can generate 72,000 time steps at 24 kHz. This is solved by using neural codecs (audio tokenizers) that compress raw audio into a dense, token-like representation, allowing the LLM to process the audio domain abstractly.
- Multi-Stream Language Models: To achieve full-duplex capability, the model must move beyond single-sequence prediction (half-duplex) and model two independent token streams simultaneously. This requires multi-stream language models.
- System Architecture Comparison: The pipeline has evolved from a constrained STT $ ightarrow$ NLU $ ightarrow$ TTS (Siri) to a single, end-to-end S2S model (absorbing all steps), and finally to a hybrid architecture (S2S interface + background text LLM).

## Practical implications

- For build engineers, the primary architectural decision is balancing the cost and complexity of end-to-end S2S models versus the optionality and intelligence of a hybrid (S2S + Text LLM) backend.
- The hybrid approach is economically superior because it avoids running complex, expensive multimodal models for simple conversational chit-chat.
- The ability to control the backend intelligence (e.g., switching the LLM provider) is a major advantage over relying solely on a single S2S model provider.

## Topics

Voice AI, Speech Recognition, Large Language Models (LLMs), Full Duplex Communication, Agentic Systems, Neural Codecs, Gradium, Moshi, Hibiki, GPT-4o

Source: https://www.youtube.com/watch?v=a8EcVumh71E