# ⏭️ Forward Deployed: Voice AI on what works in 2026

## Executive summary

The discussion explores the advanced engineering challenges of deploying real-time voice AI agents in enterprise settings. Speakers emphasize that achieving human-like performance requires complex, multi-stage cascaded pipelines (Speech-to-Text $\rightarrow$ LLM $\rightarrow$ Text-to-Speech), rather than simple end-to-end Voice-to-Voice models. Key challenges include managing latency trade-offs, ensuring reliability through model fallbacks, solving non-trivial problems like turn-taking, and architecting systems to handle complex context while maintaining low cost.

## Key takeaways

- Architectural Standard: Cascaded Pipeline: The current state-of-the-art for reliable voice agents is a cascaded pipeline (Speech-to-Text $\rightarrow$ LLM $\rightarrow$ Text-to-Speech). While Voice-to-Voice models are appealing, they currently lack the necessary reliability and control required for enterprise use cases. [0:03:45]
- The Latency vs. Intelligence Trade-off: There is an inherent trade-off between the intelligence of the response and the latency. Achieving highly intelligent responses often results in slower processing, which must be managed to feel natural to the end user. [0:03:15]
- Complexity of Conversation Flow: Solving conversational nuances like turn-taking (knowing when a pause means thinking versus ending the conversation) is not trivial. Furthermore, systems must account for different use cases: outbound calls are easier because guardrails can be strict, while inbound calls require handling unknown user context. [0:03:25]
- Optimizing LLM Context and Cost: Due to the limitations of LLMs (forgetting information in the middle of a massive prompt), advanced architectures must employ context optimization techniques. This includes using specialized models or implementing 'compaction' methods, which is becoming critical for both performance and unit economics. [0:03:58]

## Technical details

- Voice Agent Pipeline Architecture: The standard pipeline involves: (1) Voice Input $\rightarrow$ Transport (WebRTC, phone call, WebSockets); (2) Speech-to-Text (STT), potentially with background noise removal or voice isolation; (3) Turn Detection Model (using Voice Activity Detection/Smart Turn Models); (4) LLM Inference (which may generate tool calls); (5) Text-to-Speech (TTS); and (6) Streaming the output back to the user. [0:04:12]
- Model Reliability and Guardrails: To ensure reliability, systems require a 'waterfall of models' (redundancy) to handle failures from primary providers. The cascaded approach allows for enforcing rigid guardrails by passing input through supervisor models to detect prompt injection or social engineering before reaching the LLM. [0:03:45]
- Performance Optimization: Improving performance requires parallelizing steps across the pipeline (e.g., running multiple LLMs downstream from a single text output). Latency reduction is achieved by shaving off milliseconds at every stage of the process, making it feel 'snappy' despite complex background operations. [0:03:52]
- Multilingual Deployment: For multilingual support (e.g., Japan), the cascaded model is advantageous because different components (STT, TTS) can be swapped out if one piece does not perform well in a specific language or dialect. [0:04:35]

## Practical implications

- When designing a voice agent, prioritize testing and iteration over theoretical perfection; the final architecture is highly dependent on the specific customer use case.
- To manage cost and latency, consider self-hosting smaller, specialized models (like Electron) instead of relying solely on large commercial APIs, which can suffer from unpredictable latency spikes.
- For complex workflows, utilize a hybrid approach: maintain continuous voice interaction using an asynchronous model while delegating specific lookups or tasks to a separate, controlled cascaded pipeline.
- Benchmark performance rigorously by running evaluations (STT, LLM, TTS) against open-source benchmarks before live deployment.

## Topics

Voice AI, Agentic Engineering, LLMs, Conversational AI, Real-time Systems, Build Architecture, Latent Space, OpenAI APIs / Gemini 2.5

Source: https://www.youtube.com/watch?v=MwNvowwcZOo