Speech-to-Speech Model Research at Google DeepMind — Valeria Wu Fon & Tom Ouyang, Google DeepMind
Google DeepMind presented research on Speech-to-Speech models, positioning them as the foundation for the 'agentic future' of voice interaction. The core argument is that modern models must move beyond simple Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) chains. By leveraging natively multimodal pre-training (audio, video, text), these models achieve a 'trifecta' of conversational fluency, high intelligence (task completion/reasoning), and multimodality (handling video, screen shares, and documents). This enables complex, real-time applications like live multilingual translation and proactive, low-latency conversational agents.
Key takeaways
-
The Shift from Cascaded to Unified Models
2:31
Historically, speech processing required multiple hand-built components (feature extraction, acoustic modeling, language modeling, rescoring). Modern LLMs, trained on interleaved multimodal examples, collapse this chain, allowing a single model to understand and transition between audio, video, and text inputs.
-
The Three Pillars of Speech-to-Speech Models
10:01
A robust model must balance three vectors: 1) Conversational (low latency/snappy); 2) Intelligent (task completion, instruction following); and 3) Multimodal (accepting video, screen shares, and documents). Improving one vector often degrades the others (e.g., increasing intelligence can decrease time to first audio).
-
Real-Time Multilingual Translation
5:27
The model can perform streaming, real-time translation across multiple speakers and languages (e.g., English, Spanish, Italian, Chinese) with quality comparable to offline systems, a capability difficult for cascaded systems.
-
Proactive Audio and Multimodal Output
13:57
Advanced features include 'proactive audio,' where the model knows when to respond despite background noise or when another speaker is talking. Furthermore, the model can generate multimodal output, including customized real-time avatars with low-latency lip-syncing.