What's new in the Gemini Live API
Google DeepMind introduced significant updates to the Gemini Live API, enhancing real-time voice and vision interactions. Key advancements include asynchronous function calling for background tool execution, proactive audio for relevant agent responses, and context injection via `send client_content`. Most notably, the API now supports frontier-level background reasoning, allowing complex tasks (like generating SVGs) to run while the agent maintains conversational responsiveness.
Key takeaways
-
Asynchronous Function Calling
3:55
The API supports async function calling, enabling long-running tools to execute in the background while the agent continues the conversation, improving perceived latency and user experience. (See demonstration of checking an order ID 1234 XYZ)
-
Proactive Audio and Context Injection
2:00
Proactive audio ensures the agent only speaks when relevant. Context can be seamlessly injected using `send client_content` without forcing a verbal turn, useful for backchanneling information. (Demonstrated early in the video)
-
Frontier-Level Background Reasoning
5:20
Higher reasoning capabilities are now available in the Live API, allowing the model to handle complex, time-consuming tasks (e.g., drawing an SVG) in the background while remaining highly responsive to ongoing user dialogue.