# 5 Voice Agent Failure Modes You'll Hit in Week One — Venky B, Plivo

## Executive summary

This talk details five critical failure modes encountered when deploying voice AI agents from a proof-of-concept (PoC) environment to production scale. The core message emphasizes that successful production deployment requires shifting focus from end-to-end transcript processing to structured data collection, rigorous latency management, and building robust normalization layers between the LLM and Text-to-Speech (TTS) components. Key technical recommendations include using smaller, self-hosted open-source models for low latency, treating data collection as unit-testable field validation, and normalizing all output before synthesis.

## Key takeaways

- Latency and Time to First Audio (TTFA): Industry average TTFA often falls between 750ms and 1,200ms, leading to user drop-off. To improve this, the speaker recommends using smaller, self-hosted open-source models (e.g., Quen 3.5, Gemma 4) to target under 300ms, balancing cost, intelligence, and latency.
- Data Collection Must Be Structured: Instead of treating input as a single transcript, data collection should be modeled like structured fields (e.g., using Pydantic or Zod). This approach significantly increases accuracy (from ~30% to mid-90s) by allowing validation and error handling per field (e.g., phone numbers, addresses).
- Normalization Layer is Critical: Never feed raw LLM output directly to a TTS engine. A dedicated normalization layer must be implemented to strip markdown/emojis, apply custom pronunciation dictionaries, and control speaking speed (e.g., 0.7x or 0.8x) to ensure proper pronunciation of entities.
- Transcription is Brittle by Default: Transcription engines struggle with proper nouns, jargon, and code-switched languages. Solutions include implementing dynamic keyword boosting and post-processing the raw transcript using an LLM layer, which provides necessary domain context.

## Technical details

- Model Selection and Sizing: For latency-sensitive applications, open-source models are recommended. For multilingual support, Gemma 4 is noted as significantly better than Quen 3.5 due to superior token fertility. Mixture of Experts (MoE) models (3B-4B) are generally sufficient, but for deep fine-tuning, an 8B or 12B model is the minimum requirement.
- Data Validation and Testing: Evaluation (evals) must treat collected fields as unit tests, rather than running end-to-end agent test cases. This ensures reliability and repeatability in field collection.
- Multilingual Normalization: To handle code-switched languages or non-Latin scripts, the system must use a normalization layer, such as an NLM or neural transliteration engine, to ensure consistent input to the LLM.

## Practical implications

- Implement structured data models (e.g., Pydantic) for all input fields to move beyond end-to-end transcript processing.
- Build a dedicated normalization layer between the LLM and TTS to manage formatting, pronunciation, and entity handling.
- Prioritize low latency (sub-300ms TTFA) by selecting appropriately sized, self-hosted open-source models.
- Shift testing methodology to field-level unit tests rather than relying solely on end-to-end test cases.

## Topics

Voice AI, LLM Orchestration, Telephony APIs, Data Validation, Speech Synthesis, Plivo, Quen 3.5, Gemma 4

Source: https://www.youtube.com/watch?v=vblnYHzBgS4