# "My name is... my name is...": A Linguistic Map for Voice Agents — Midam Kim, ServiceNow

## Executive summary

Midam Kim presents a linguistic framework for diagnosing failures in voice AI, arguing that these failures are not isolated bugs but structured issues. She proposes that human communication is a 'joint activity' involving the continuous updating of a 'mental model.' The framework maps this process onto two channels (listening and speaking) and four interdependent levels: sounds, words, interaction, and mental model. Successful voice AI requires holistic orchestration across all these layers, rather than optimizing components (like ASR or TTS) in isolation.

## Key takeaways

- Voice AI is a Joint Activity: Human communication is a joint activity where both parties contribute sounds and words, continuously updating a shared mental model. Voice AI systems must replicate this joint nature to be effective.
- The Linguistic Framework: The system must be analyzed across two channels (listening/speaking) and four interdependent levels: sounds, words, interaction, and mental model. Failure in one area (e.g., STT failure at the sound level) impacts the entire system.
- Mental Model Accumulation: Unlike text chat where history remains visible, in voice interactions, sounds and words vanish. The only persistent element that matters for user satisfaction is the user's accumulating mental model.
- System Adaptability is Key: The system must be designed to be dynamic, adapting to context, emotion, and language change over the course of the call, rather than functioning as a static pipeline.

## Technical details

- Linguistic Framework Components: The framework consists of two channels (listening and speaking) and four levels: 1) Sounds (e.g., ASR/STT accuracy); 2) Words (understanding vocabulary); 3) Interaction (turn-taking/timing); and 4) Mental Model (tracking user intent). These components are interdependent; improving sounds requires considering the word level, and so on.
- Voice AI Failure Modes: Observed failures include: 1) Misrecognition (e.g., bot hearing 'Midan' instead of 'Midam'); 2) STT failure (cutting off the user while reading an account number); 3) Lack of context retention (asking for repeated information without tracking the mental model); and 4) Failure to initiate interactive clarification.
- System Optimization Focus: Improvements must go beyond optimizing individual components (ASR/TTS). Focus areas include: good turn-to-turn detection, latency management, emotion detection, context retention, and dynamic orchestration.

## Practical implications

- Shift the design focus from optimizing individual ML components (ASR, TTS) to orchestrating the entire linguistic and cognitive experience.
- Implement robust mechanisms for tracking and updating the user's mental model throughout the conversation timeline.
- Design the system to be dynamically adaptive, anticipating changes in user state, emotion, and language over time.
- Use the linguistic framework to diagnose system weaknesses, recognizing that failure is a structural, not random, event.

## Topics

Voice AI, Linguistics, Natural Language Processing (NLP), Speech Recognition, System Architecture, User Experience (UX), Eva bench

Source: https://www.youtube.com/watch?v=IDNfAZVKvPE