# Build Smarter Voice Agents

## Executive summary

This panel discussion provides deep insights into building production-grade voice AI agents. Key architectural recommendations favor the deconstructed cascading pipeline (ASR $\rightarrow$ LLM $\rightarrow$ TTS) due to its superior flexibility for optimization and model swapping. Engineers must prioritize managing latency within a 1–1.5 second budget, implementing robust fallback systems across all stack components (ASR, LLM, TTS), and utilizing advanced context management techniques like 'Scratchpads' to maintain conversational continuity over long interactions.

## Key takeaways

- Architectural Choice: Cascading Pipeline: The cascading architecture is preferred because it allows for individual optimization of the ASR, LLM, and TTS layers. This modularity provides greater flexibility than a full Speech-to-Speech (S2S) stack when integrating new models or optimizing specific components.
- Latency Management: The 'golden metric' for voice agent response time is between 1 to 1.5 seconds. Exceeding this budget can be unnerving for users, making latency a primary design constraint over pure accuracy in many cases.
- Context and Memory Management: To prevent negative user sentiment from repeating information, agents must implement context stores (e.g., 'Scratchpads') to track customer profiles, preferences, and key facts across multiple turns or sessions.

## Technical details

- Voice Agent Architecture: The standard approach is a cascading pipeline: ASR $\rightarrow$ LLM $\rightarrow$ TTS. While S2S exists, the modularity of the cascaded stack allows for better control and optimization at each node.
- Evaluation Metrics (Evals): ASR evaluation focuses on Word Error Rate (WER), with special attention paid to errors in numbers. LLM evaluation uses internal benchmarks over difficult scenarios, often judged by another LLM. TTS is highly subjective and requires human 'VIP checks.'
- Resilience and Fallbacks: Production systems must implement fallbacks for every component (ASR, LLM, TTS). Instead of running providers in parallel (due to cost), monitoring jobs detect failures and trigger a hard cutover to an alternative provider. Latency routing is also used across multiple LLM providers (e.g., OpenAI, Azure) to ensure the lowest possible latency.
- Advanced Concepts: Loop Engineering: Loop engineering suggests moving beyond prompt iteration by defining measurable metrics that an agent can iteratively improve against itself during development, enabling self-evaluating agents.

## Practical implications

- Prioritize modularity in your voice stack to allow for independent optimization and swapping of components (ASR, LLM, TTS).
- Implement robust monitoring and fallback mechanisms across all third-party APIs to ensure service continuity.
- Focus on context persistence by building a structured data layer (like a 'Scratchpad') to track user information over time, mitigating the negative impact of repetitive questioning.
- Measure performance not just on accuracy, but critically on latency (aiming for 1–1.5 seconds).

## Topics

Voice AI, LLMs, Natural Language Processing, System Architecture, Build Engineering, Latency Optimization, AssemblyAI, hiresuper.com, https://www.assemblyai.com

Source: https://www.youtube.com/watch?v=yqm1untg9xY