Voice Agents Can Just Do Things — Charlie Guo, OpenAI
The presentation challenges the common misconception that voice agents must respond using speech. Instead, the speaker argues that voice models can utilize three distinct, non-mutually exclusive modes: Speech-to-Speech, Speech-to-Action, and Event-to-Speech. For developers, the key takeaway is that building voice agents is simplified by recognizing that existing application verbs (API endpoints, React hooks) can be directly exposed as tools for the model to call. Furthermore, the talk details the technical shift toward native audio processing (audio in, audio out) and introduces advanced models like GPT-realtime 2, which adds reasoning and structured tool calling capabilities.
Key takeaways
-
The Three Modes of Voice Interaction
0:12
Voice interaction is categorized into three modes: Speech-to-Speech (e.g., coaching, translation), Speech-to-Action (user talks, model uses tools, e.g., form filling), and Event-to-Speech (model reacts to an event, e.g., proactive alerts).
-
Developer Focus: Exposing Verbs as Tools
10:58
Developers can integrate voice by treating existing application verbs (API endpoints, React hooks) as callable tools, allowing the model to drive the existing software rather than just generating text.
-
The Shift to Native Audio Processing
Modern voice models are moving away from the chained approach (transcribe speech -> LLM -> text -> audio) to native audio tokens, which preserves critical context like tone, cadence, and emotional impact.
-
GPT-realtime 2 Capabilities
The latest model in the real-time family offers reasoning capabilities, parallel tool calling, and 'preambles' to manage user expectations while actions are performed in the background.