Tolan: Voice-First AI Companion — Paula Dozsa, Tolan
Summary
Paula Dozsa, an engineer on the Tolan team, details the unique engineering challenges of building a voice-first AI companion compared to traditional text-based LLM applications. She emphasizes that voice introduces 'conversational volatility' (fast turns, interruptions) which requires fundamental shifts in pipeline design, including smart turn detection, tiered model routing based on emotional stakes, and rebuilding context per turn rather than relying on a continuous cache. The talk also covers using advanced AI agents (like Claude) to build the product itself, achieving significant improvements in stability and feature development.
Key takeaways
-
Voice vs. Text LLM Assumptions
4:05
Text chat assumes slow turns and stable context, while voice is characterized by fast turns and volatile context (interruptions, subject changes mid-sentence). This volatility requires building for messy, real-world speech patterns.
-
Optimizing for Interruptions
5:38
Instead of minimizing interruptions, the team focused on building smart turn-taking that reads speech patterns to prevent early, incorrect agent interventions. They paid an extra 60 milliseconds of latency to achieve this.
-
Memory as Retrieval System
12:32
To handle volatile context, memory is treated as a retrieval system, not a transcript. Facts and preferences are embedded, stored in a vector database (with sub-50ms lookups), and compressed nightly to resolve contradictions and merge duplicates.
-
Context Reassembly
Context must be reassembled from parts every single turn (summary, user persona, retrieved memories, tone guidance) because reusing old context is a 'trap' when the user pivots subjects.
Technical details
-
Pipeline Measurement
520s
Latency is measured across every stage: end-of-utterance detection, transcription, time to first token (often ~1 second), model generation, and text-to-speech playback. The biggest quality jump came from moving to GPT-5.1 on the responses API.
-
Tiered Model Routing
612s
The system uses a 'tone router' (a cheap classifier) to route turns to different models. High-stakes moments (e.g., first conversation, crisis tones) use a 'frontier model,' while lighter turns use smaller, faster models. This maximizes unit economics by ensuring the high-cost model is only used when necessary.
-
AI Agent Development Workflow
The team uses a fleet of agents for development: implementation agents build code, separate review agents enforce standards, and a PR shepherd watches pull requests, iterating against CI failures and review comments. Triage bots can reconstruct crashes and open PRs to fix bugs.
-
Evaluation and Iteration
The process includes 'eval' cycles where agents map personality traits onto the codebase, generating a 'voice Bible.' Changes are then evaluated against production logs over multiple 'find fix verify' rounds.
Mentioned resources
- Tolan
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.