"My name is... my name is...": A Linguistic Map for Voice Agents — Midam Kim, ServiceNow
Midam Kim presents a linguistic framework for diagnosing failures in voice AI, arguing that these failures are not isolated bugs but structured issues. She proposes that human communication is a 'joint activity' involving the continuous updating of a 'mental model.' The framework maps this process onto two channels (listening and speaking) and four interdependent levels: sounds, words, interaction, and mental model. Successful voice AI requires holistic orchestration across all these layers, rather than optimizing components (like ASR or TTS) in isolation.
Key takeaways
-
Voice AI is a Joint Activity
5:00
Human communication is a joint activity where both parties contribute sounds and words, continuously updating a shared mental model. Voice AI systems must replicate this joint nature to be effective.
-
The Linguistic Framework
11:54
The system must be analyzed across two channels (listening/speaking) and four interdependent levels: sounds, words, interaction, and mental model. Failure in one area (e.g., STT failure at the sound level) impacts the entire system.
-
Mental Model Accumulation
Unlike text chat where history remains visible, in voice interactions, sounds and words vanish. The only persistent element that matters for user satisfaction is the user's accumulating mental model.
-
System Adaptability is Key
The system must be designed to be dynamic, adapting to context, emotion, and language change over the course of the call, rather than functioning as a static pipeline.