Topic

Interactions API

All digests tagged Interactions API

Agents Without Code: Skills, YAML, and Filesystems Replaced Python — Philipp Schmid, Google DeepMind thumbnail

· 18:28

Agents Without Code: Skills, YAML, and Filesystems Replaced Python — Philipp Schmid, Google DeepMind

The presentation details the evolution of LLM agents, demonstrating a shift from complex, brittle Python code loops to declarative, file-based definitions using system instructions and skills. The speaker shows that modern agent architectures, such as the Gemini API's anti-gravity agent, utilize a hosted sandbox and network proxy to manage state and credentials securely. This allows agents to operate using general-purpose tools (like GitHub CLI or Google Search) defined in files (e.g., `AGENTS.md`), drastically reducing the need for thousands of lines of custom orchestration code.

Key takeaways

  1. The Agent Evolution: Code to Files

    Agent development is moving away from writing explicit Python loops, JSON schemas, and tool routing logic. The core functionality is now expressed in files (Markdown/Skills) that define instructions, rules, and capabilities, allowing the model to use general tools.

  2. Server-Side State Management 14:02

    The new architecture handles complex tasks by moving loops, tool routing, session state, and context compaction to the server side, requiring only a single API call with new inputs.

  3. Security and Isolation 12:32

    A hosted sandbox and network proxy ensure that the agent never sees the actual credentials, injecting tokens only when outbound requests are made, and allowing domain restriction for enhanced security.

  4. Focus on Domain Logic 16:40

    The primary work for developers is now defining the domain instructions, rules, and evaluation criteria (the 'what'), rather than writing the infrastructure code (the 'how').

Watch on YouTube Full article

Build voice-first apps with Gemini 3.5 Transcribe thumbnail

· 1:44

Build voice-first apps with Gemini 3.5 Transcribe

Google launched Gemini 3.5 Transcribe, an advanced LLM-based model designed for building voice-first applications. This model is available via both the Interactions API and the Live API, offering fast, contextually accurate transcription of multi-speaker recordings. Key strengths include superior recognition of structured data like email addresses and phone numbers, as well as robust support for over 70 different languages.

Key takeaways

  1. Model Availability

    Gemini 3.5 Transcribe is available on both the Interactions API and the Live API.

  2. Structured Data Recognition

    The LLM-based model excels at transcribing alphanumerics, such as email addresses (e.g., thorwebdev@google.com) and recognizing correct US phone number formats.

  3. Multi-Language Support

    The model can recognize and transcribe over 70 different languages, even when language hints are set to English.

Watch on YouTube Full article

How to build with Gemini 3.5 Transcribe thumbnail

· 4:50

How to build with Gemini 3.5 Transcribe

Google DeepMind launched Gemini 3.5 Transcribe, an LLM-based transcription model available via both the Interactions API and the Live API. This model significantly enhances accuracy by correctly transcribing complex data types—such as email addresses, phone numbers, and mixed units of measurement—and maintaining high performance across over 85 supported languages, even when language codes are set to English.

Key takeaways

  1. LLM-Based Transcription Model

    The model's LLM foundation allows it to handle complex data structures and context better than traditional transcription models. For example, it can correctly identify and edit email addresses even if spoken phonetically (e.g., 'tosten at google.com').

  2. Handling Complex Data Types 2:00

    Gemini 3.5 Transcribe accurately recognizes specific formats, including US phone numbers and international variations (e.g., Singapore's 8-digit format). It can also correctly interpret units of measure (e.g., meters vs. centimeters).

  3. Multilingual and Customization Support 0:40

    The model supports over 85 languages, automatically recognizing spoken language even if language hints are set to English. Accuracy can be further improved by providing custom vocabulary (e.g., names of people in a meeting) or setting specific language codes.

Watch on YouTube Full article

Access all Gemini models with the Interactions API thumbnail

· 1:24

Access all Gemini models with the Interactions API

The Gemini Interactions API is now generally available, providing a unified, single-interface gateway for accessing all Gemini models and agents. This stateful, agent-first ecosystem allows developers to build complex AI applications with persistent memory by passing previous interaction IDs, supporting multimodal inputs like voice notes and images.

Key takeaways

  1. Interactions API General Availability

    The Interactions API is now generally available for use, simplifying access to Gemini models.

  2. Unified Model Access

    It serves as a single interface gateway for all Gemini models and agents, eliminating complex AI workflow setups.

  3. Stateful Memory Support

    The API is stateful, allowing applications to build upon previous interactions by passing the prior interaction ID.

Watch on YouTube Full article

Get started with the Interactions API thumbnail

· 8:51

Get started with the Interactions API

The Interactions API is now generally available and serves as a unified interface for interacting with various Gemini models and agents. It simplifies complex interactions—ranging from synchronous model calls to multi-step agent workflows in remote environments—by providing a single API endpoint. The API supports stateful operations by allowing users to pass the previous interaction ID, enabling iterative refinement of outputs (e.g., image editing) without re-uploading source data.

Key takeaways

  1. Unified Model Access 0:10

    The Interactions API provides easy access to all Gemini models and agents through a single interface, simplifying development across various Google AI capabilities.

  2. Stateful Interactions 4:05

    The API is stateful; developers can pass the previous interaction ID to build upon prior results, which is crucial for iterative processes like image editing or complex data processing.

  3. Advanced Capabilities 0:18

    New features include managed agents, background execution tool improvements, Deep Research, and media generation (e.g., Nano banana for images, Lyriq 3 for music, TTS for speech).

Watch on YouTube Full article