Act, Confirm, or Stop? Smarter behavior for AI assistants, wearables & robots — Amit Desai, Roku
Summary
The presentation argues that improving voice AI user experience requires focusing on a second, often neglected dimension: system behavior under uncertainty. While increasing accuracy (Knob One) is critical, the system's ability to intelligently decide what to do when it is unsure (Knob Two) can yield greater user satisfaction. This is quantified using the Outcome User Cost Heuristic (OUCH), which minimizes the total user effort by assigning differential costs to various bad outcomes (e.g., playing the wrong song vs. simply stating 'I did not understand').
Key takeaways
-
The Two Knobs of Voice AI Improvement
0:03
User satisfaction can be improved by increasing technical accuracy (Knob One) or by optimizing the system's decision-making process when confidence is low (Knob Two). The latter is often overlooked.
-
The Outcome User Cost Heuristic (OUCH)
0:10
Instead of treating all errors equally, OUCH minimizes the total user cost by quantifying the relative pain of different bad outcomes (e.g., the effort required to stop a wrong song vs. the time taken to hear 'Sorry, I did not understand').
-
Adding Conversational Behavior
0:13
Introducing a third behavior—confirming the guess out loud (e.g., 'Did you mean ABC?')—splits the confidence range into three regions (Stop, Confirm, Act) and further lowers the overall user cost.
Technical details
-
System Behavior Optimization
6s
The system's decision to act, stop, or confirm is determined by setting thresholds ($T$) on a single confidence score (0 to 1). The goal is to find the optimal $T$ that minimizes the total user cost function (OUCH).
-
Cost Function Modeling
10s
The total user cost is calculated by summing the unit costs of bad outcomes (e.g., 10 seconds for a wrong song, 4 seconds for 'I did not understand') multiplied by the number of times that outcome occurs across the data distribution.
-
Multimodal Extension
11s
The OUCH principle scales to multimodal interfaces (like TVs). Instead of speech confirmation, the system can display choices on screen, changing the variables and unit costs in the optimization equation.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.