# The next generation of voice AI with Google DeepMind and Sierra AI

## Executive summary

The discussion outlines the evolution of voice AI from traditional pipelines to advanced native audio models, focusing on achieving truly real-time, conversational experiences. Key advancements include offering specialized models (lightweight for speed, enterprise for precision), improving metrics beyond Word Error Rate (WER) to measure conversational flow, and enabling seamless multilingual code-switching and complex, multi-step agentic tasks.

## Key takeaways

- Dual Model Architecture: Developers now have two options: a lightweight, faster model for quick conversations, and a more robust, enterprise-grade model designed for high-stakes environments requiring multi-step function calling and high accuracy.
- Advanced Latency Metrics: Conversational quality is measured by two critical latencies: Time to First Audio (TFA) and Time to First Useful Response (TFUR), both of which must be minimized to maintain a natural, uninterrupted dialogue flow.
- Multilingual Code-Switching: Native audio models are highly effective at understanding and navigating language shifts and mixed-language phrasing, avoiding the 'broken telephone' effect common in traditional text-based transcription setups.

## Technical details

- Conversational Latency Benchmarking: The goal is to move beyond static context multi-turn benchmarks. Key metrics include minimizing Time to First Audio (TFA) and Time to First Useful Response (TFUR). The industry is moving toward user simulator benchmarks that incorporate real noise and diverse personas.
- Agentic Voice Tasks: The use case for agents (e.g., outbound sales agents) requires long-horizon reasoning and complex orchestration, necessitating models capable of multi-step function calling and stateful task management while maintaining a natural vocal flow.
- Voice Quality Hierarchy: Voice quality is viewed in a hierarchy: 1) Accuracy (Can the job be done?), 2) Quality (Is the call tolerable?), and 3) Experience (Is it pleasant and natural to listen to?).

## Practical implications

- Build engineers must adapt benchmarking strategies to measure dynamic metrics like conversational flow and turn-taking, rather than relying solely on static metrics like WER.
- Designing voice agents requires careful consideration of latency trade-offs, prioritizing both Time to First Audio and Time to First Useful Response.
- The architecture must support complex stateful task orchestration and reliable multi-step function calling to handle real-world, high-stakes interactions.

## Topics

Conversational AI, Native Audio Models, Real-Time Systems, Agentic Workflow, Benchmarking, Google DeepMind, Sierra AI, TAU

Source: https://www.youtube.com/watch?v=-G431WC_-RM