Tolan: Voice-First AI Companion — Paula Dozsa, Tolan
Paula Dozsa, an engineer on the Tolan team, details the unique engineering challenges of building a voice-first AI companion compared to traditional text-based LLM applications. She emphasizes that voice introduces 'conversational volatility' (fast turns, interruptions) which requires fundamental shifts in pipeline design, including smart turn detection, tiered model routing based on emotional stakes, and rebuilding context per turn rather than relying on a continuous cache. The talk also covers using advanced AI agents (like Claude) to build the product itself, achieving significant improvements in stability and feature development.
Key takeaways
-
Voice vs. Text LLM Assumptions
4:05
Text chat assumes slow turns and stable context, while voice is characterized by fast turns and volatile context (interruptions, subject changes mid-sentence). This volatility requires building for messy, real-world speech patterns.
-
Optimizing for Interruptions
5:38
Instead of minimizing interruptions, the team focused on building smart turn-taking that reads speech patterns to prevent early, incorrect agent interventions. They paid an extra 60 milliseconds of latency to achieve this.
-
Memory as Retrieval System
12:32
To handle volatile context, memory is treated as a retrieval system, not a transcript. Facts and preferences are embedded, stored in a vector database (with sub-50ms lookups), and compressed nightly to resolve contradictions and merge duplicates.
-
Context Reassembly
Context must be reassembled from parts every single turn (summary, user persona, retrieved memories, tone guidance) because reusing old context is a 'trap' when the user pivots subjects.