# Speech-to-Speech Model Research at Google DeepMind — Valeria Wu Fon & Tom Ouyang, Google DeepMind

## Executive summary

Google DeepMind presented research on Speech-to-Speech models, positioning them as the foundation for the 'agentic future' of voice interaction. The core argument is that modern models must move beyond simple Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) chains. By leveraging natively multimodal pre-training (audio, video, text), these models achieve a 'trifecta' of conversational fluency, high intelligence (task completion/reasoning), and multimodality (handling video, screen shares, and documents). This enables complex, real-time applications like live multilingual translation and proactive, low-latency conversational agents.

## Key takeaways

- The Shift from Cascaded to Unified Models: Historically, speech processing required multiple hand-built components (feature extraction, acoustic modeling, language modeling, rescoring). Modern LLMs, trained on interleaved multimodal examples, collapse this chain, allowing a single model to understand and transition between audio, video, and text inputs.
- The Three Pillars of Speech-to-Speech Models: A robust model must balance three vectors: 1) Conversational (low latency/snappy); 2) Intelligent (task completion, instruction following); and 3) Multimodal (accepting video, screen shares, and documents). Improving one vector often degrades the others (e.g., increasing intelligence can decrease time to first audio).
- Real-Time Multilingual Translation: The model can perform streaming, real-time translation across multiple speakers and languages (e.g., English, Spanish, Italian, Chinese) with quality comparable to offline systems, a capability difficult for cascaded systems.
- Proactive Audio and Multimodal Output: Advanced features include 'proactive audio,' where the model knows when to respond despite background noise or when another speaker is talking. Furthermore, the model can generate multimodal output, including customized real-time avatars with low-latency lip-syncing.

## Technical details

- Pre-2018 Speech Pipeline: The traditional pipeline involved separate components: feature extraction, acoustic modeling, pronunciation modeling, language modeling, and a rescoring pass, limiting the system's ability to handle tone, emotion, or images.
- Natively Multimodal Pre-training: Gemini models are trained on interleaved multimodal examples (text, video, audio) in the pre-training phase. This teaches the model to understand the relationship between different data types, enabling tasks like summarizing a video while annotating timestamps.
- Model Architecture: The goal is to create a single, unified token embedding space that handles ASR, TTS, and agentic tasks simultaneously, making the system highly 'promptable' and versatile.

## Practical implications

- Developing universal voice agents for enterprise use cases (e.g., cloud APIs, voice agent deployments).
- Improving customer service interactions through roadside assistance agents that can process alphanumeric data (registration plates, postcodes).
- Enabling complex info-seeking tasks (e.g., describing a sofa) that require multimodal input and multilingual, culturally aware output (e.g., retaining 'mid-century' in Spanish).
- Facilitating seamless, low-latency communication in noisy, real-world environments (e.g., on the go, on a train).

## Topics

Artificial Intelligence (AI), Large Language Models (LLMs), Speech Recognition (ASR), Speech Synthesis (TTS), Multimodality, Build Engineering, Gemini

Source: https://www.youtube.com/watch?v=18Um2VjMM_g