# Tolan: Voice-First AI Companion — Paula Dozsa, Tolan

## Executive summary

Paula Dozsa, an engineer on the Tolan team, details the unique engineering challenges of building a voice-first AI companion compared to traditional text-based LLM applications. She emphasizes that voice introduces 'conversational volatility' (fast turns, interruptions) which requires fundamental shifts in pipeline design, including smart turn detection, tiered model routing based on emotional stakes, and rebuilding context per turn rather than relying on a continuous cache. The talk also covers using advanced AI agents (like Claude) to build the product itself, achieving significant improvements in stability and feature development.

## Key takeaways

- Voice vs. Text LLM Assumptions: Text chat assumes slow turns and stable context, while voice is characterized by fast turns and volatile context (interruptions, subject changes mid-sentence). This volatility requires building for messy, real-world speech patterns.
- Optimizing for Interruptions: Instead of minimizing interruptions, the team focused on building smart turn-taking that reads speech patterns to prevent early, incorrect agent interventions. They paid an extra 60 milliseconds of latency to achieve this.
- Memory as Retrieval System: To handle volatile context, memory is treated as a retrieval system, not a transcript. Facts and preferences are embedded, stored in a vector database (with sub-50ms lookups), and compressed nightly to resolve contradictions and merge duplicates.
- Context Reassembly: Context must be reassembled from parts every single turn (summary, user persona, retrieved memories, tone guidance) because reusing old context is a 'trap' when the user pivots subjects.

## Technical details

- Pipeline Measurement: Latency is measured across every stage: end-of-utterance detection, transcription, time to first token (often ~1 second), model generation, and text-to-speech playback. The biggest quality jump came from moving to GPT-5.1 on the responses API.
- Tiered Model Routing: The system uses a 'tone router' (a cheap classifier) to route turns to different models. High-stakes moments (e.g., first conversation, crisis tones) use a 'frontier model,' while lighter turns use smaller, faster models. This maximizes unit economics by ensuring the high-cost model is only used when necessary.
- AI Agent Development Workflow: The team uses a fleet of agents for development: implementation agents build code, separate review agents enforce standards, and a PR shepherd watches pull requests, iterating against CI failures and review comments. Triage bots can reconstruct crashes and open PRs to fix bugs.
- Evaluation and Iteration: The process includes 'eval' cycles where agents map personality traits onto the codebase, generating a 'voice Bible.' Changes are then evaluated against production logs over multiple 'find fix verify' rounds.

## Practical implications

- Building voice-first AI requires designing for conversational volatility, treating speech as inherently messy and interrupt-prone.
- The unit economics of LLM applications must account for turn-by-turn model routing based on the emotional stakes, not just the cost.
- Context management must shift from caching history to actively reassembling context from multiple, discrete parts for every turn to prevent drift.
- AI agents can be used not only for content generation but also for core engineering tasks like bug fixing, code review, and feature evaluation.

## Topics

Voice AI, LLM Architecture, Conversational Design, AI Agents, Build Engineering, Natural Language Processing, Tolan

Source: https://www.youtube.com/watch?v=xLUQOqjudtA