# Voice Agents Can Just Do Things — Charlie Guo, OpenAI

## Executive summary

The presentation challenges the common misconception that voice agents must respond using speech. Instead, the speaker argues that voice models can utilize three distinct, non-mutually exclusive modes: Speech-to-Speech, Speech-to-Action, and Event-to-Speech. For developers, the key takeaway is that building voice agents is simplified by recognizing that existing application verbs (API endpoints, React hooks) can be directly exposed as tools for the model to call. Furthermore, the talk details the technical shift toward native audio processing (audio in, audio out) and introduces advanced models like GPT-realtime 2, which adds reasoning and structured tool calling capabilities.

## Key takeaways

- The Three Modes of Voice Interaction: Voice interaction is categorized into three modes: Speech-to-Speech (e.g., coaching, translation), Speech-to-Action (user talks, model uses tools, e.g., form filling), and Event-to-Speech (model reacts to an event, e.g., proactive alerts).
- Developer Focus: Exposing Verbs as Tools: Developers can integrate voice by treating existing application verbs (API endpoints, React hooks) as callable tools, allowing the model to drive the existing software rather than just generating text.
- The Shift to Native Audio Processing: Modern voice models are moving away from the chained approach (transcribe speech -> LLM -> text -> audio) to native audio tokens, which preserves critical context like tone, cadence, and emotional impact.
- GPT-realtime 2 Capabilities: The latest model in the real-time family offers reasoning capabilities, parallel tool calling, and 'preambles' to manage user expectations while actions are performed in the background.

## Technical details

- Speech-to-Action Use Cases: Applications include automating complex tasks like filling out government forms (talking for minutes vs. hours of typing) and guiding creative processes (e.g., composing music or painting) when the user's taste exceeds their operational ability.
- System Architecture: The ideal system combines all three modes. The underlying technical shift involves moving from turn-based abstraction to continuous streaming audio in and out.
- Model Enhancements: GPT-realtime 2 provides reasoning to the audio medium, allowing the model to 'think' before speaking. It also supports parallel tool calling and 'preambles' to communicate background processing to the user.

## Practical implications

- Developers should shift their focus from 'what kind of voice agent' to 'what is the role of voice and audio' in the interaction.
- The most effective voice experiences will combine all three modes (Speech-to-Speech, Speech-to-Action, Event-to-Speech).
- For building, focus on exposing existing application logic (API endpoints, hooks) as callable tools.
- Implement guardrails and safety checks, even when integrating voice, as the core functionality remains the existing software.
- Consider accessibility: Voice can be a critical tool for users who lose mobility or dexterity.
- Use 'preambles' when tool calling to manage user expectations and reduce perceived latency.

## Topics

AI, Voice UX, Tooling, Developer Experience, LLMs, OpenAI, GPT-realtime 2

Source: https://www.youtube.com/watch?v=OpY6MmZFeHo