Building the Voice OS at Willow
This session details the engineering journey of building a Voice OS (Willow Voice) and expanding into advanced AI communication tools (Willow Scribe). The core challenge shifts from achieving accurate Speech Recognition (ASR) to making the output fast, reliable, and highly personalized. Key technical lessons include the necessity of low latency for user adoption, the use of LLMs for structuring unstructured human output (e.g., emails, Slack messages), and employing advanced techniques like Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to infer user intent and style while maintaining user privacy.
Key takeaways
-
Latency is a primary adoption hurdle.
22:30
While quantitative speed (e.g., typing vs. dictation) is important, the perceived speed (low latency) is critical. The initial delay in the Willow demo (3-5 seconds) caused users to revert to typing, demonstrating that immediate feedback is a top priority for product adoption. (Timestamp: 13:50)
-
AI communication requires inferring style and tone.
27:10
The challenge is moving beyond mere transcription to structuring the output (e.g., formal emails vs. casual Slack messages). This requires the model to infer the appropriate 'semi-natural language' (semi-linguistic style), which is highly variable based on context and user demographics. (Timestamp: 16:30)
-
Privacy-preserving personalization is achievable.
Personalization can be achieved without accessing semantic content by tracking limited, objective signals on the client side. By calculating metrics like the word-level Levenshtein distance (insertions, deletions, replacements) between the original and edited text, the system can infer user intent (e.g., 'deletion suggests the model was too verbose'). (Timestamp: 21:50)
-
The future of work involves automating coordination.
The speaker predicts that the majority of engineering time will shift from execution to coordination, defining requirements, and communicating. Future tools must act as sophisticated executive assistants, capable of understanding context and coordinating actions across people and agents. (Timestamp: 26:30)