The Video Signal

Topic

Multimodal AI

All digests tagged Multimodal AI

New frontier AI models, TypeSafe’s Jev AI, & NASA’s IBM collab thumbnail

· 39:24

New frontier AI models, TypeSafe’s Jev AI, & NASA’s IBM collab

The AI landscape is rapidly shifting toward efficiency and specialized, structured decision-making. The discussion covered the current model release cycle, noting that models like Anthropic's Claude Opus 5.5 and OpenAI's GPT-6 Sol are achieving high performance at significantly reduced token costs. A major focus was placed on TypeSafe's Jev AI, a 'System One' model designed to replace verbose text generation with fast, structured, and calibrated decisions, which is highly beneficial for enterprise automation. Finally, the session highlighted the collaborative use of foundation models, such as the NASA/IBM Lunar Foundation Model, for specialized scientific tasks like crater mapping and ice detection, demonstrating AI's value beyond consumer chatbots.

Key takeaways

  1. Model efficiency is the primary trend in AI releases. 18:30

    New frontier models are prioritizing efficiency (cost and compute) over sheer size, allowing high performance at reduced token costs. This shift is driven by the need to manage compute costs and is moving the focus from model intelligence to system intelligence (agentic workflows).

  2. Jev AI enables structured, calibrated decisions. 28:50

    Jev AI, a 'System One' model, is designed to output structured decisions and probabilities directly, bypassing the need for verbose, free-form text generation. This approach is more efficient for automation and reduces the engineering complexity of parsing LLM output.

  3. Foundation models are expanding into scientific domains.

    The NASA/IBM Lunar Foundation Model demonstrates that foundation models can be effectively applied to specialized scientific data (e.g., lunar image tiles) for tasks like crater mapping and identifying polar ice, proving their value in data-intensive, non-chatbot applications.

Watch on YouTube Full article

From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI thumbnail

· 20:42

From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI

The presentation outlines the shift from traditional Vision-Language Models (VLMs) and Vision-Action Models (VALAs) toward Embodied Foundation Models. The core goal is to create unified AI capable of perceiving, reasoning, and acting in the physical world. Key breakthroughs include addressing the massive data inefficiency of long video inputs (token waste) and managing context bloat using data sparse Mixture of Experts (MoE). A major finding is a new scaling law: joint training on perception, reasoning, and control allows a 10x reduction in expensive teleoperation data by increasing video pretraining data.

Key takeaways

  1. Embodied Foundation Models (EFMs)

    The north star is building physical AI foundations that unify perception, reasoning, and control into a single model, bridging the physical and digital world. This moves beyond the limitations of separate VLM, VLA, and World Model architectures.

  2. Addressing Data Sparsity and Token Waste 5:40

    When modeling long video (e.g., one hour), millions of visual tokens are generated, but ground truth is extremely sparse (e.g., only 2% of tokens are useful). The solution involves developing a 'natural perceptive objective' to automatically predict and focus on percepts that matter (e.g., a gripper tip) rather than treating all background pixels equally.

  3. Context Management via Data Sparse MoE 9:01

    To combat context bloat from always-on cameras, the approach utilizes a data sparse Mixture of Experts (MoE) architecture. This includes a router that determines per layer which tokens to read and which to skip, allowing the model to focus compute on task-relevant information.

  4. New Scaling Law for Robotics 9:01

    Jointly training on perception, reasoning, and control yields a significant scaling law: 10x more video pretraining data can substitute for 10x less expensive teleoperation (teleop) data, drastically lowering the cost of embodied AI.

Watch on YouTube Full article

Gemma 4 12B: The Encoder-Free Model Explained thumbnail

· 2:27

Gemma 4 12B: The Encoder-Free Model Explained

Gemma 4 12B is highlighted as a unique, encoder-free model within the Gemma 4 family, designed to process audio and image inputs without the traditional overhead of dedicated encoders. This architecture significantly improves efficiency by allowing the Large Language Model (LLM) to directly process sequential or embedded information, thereby reducing the time-to-first-token compared to models that rely on large, separate vision or audio encoders.

Key takeaways

  1. Encoder-Free Multimodal Processing

    Unlike other Gemma 4 models that use a vision encoder (up to 550M parameters) and an audio encoder (Conformer, 300M parameters), Gemma 4 12B processes audio and image inputs directly, eliminating the need for these large, separate encoders.

  2. Audio Input Streamlining

    For audio, Gemma 4 12B bypasses the Conformer by simply cutting audio segments to create sets of amplitude values, which are then projected directly to the LLM because they are considered sequential information.

  3. Image Input Optimization

    Image processing uses a small embedder (35M parameters) instead of a full encoder. This embedder processes the 3D pixels and adds crucial positional information (X and Y coordinates) before the LLM takes over the task of interpreting the image meaning.

Watch on YouTube Full article

Understand the Gemma 4 model family thumbnail

· 2:21

Understand the Gemma 4 model family

The Gemma 4 family is a set of multimodal, open-source large language models available in five sizes across four architectures. The models range from efficient, dense, on-device options (E2B, E4B) utilizing Per-Layer Embeddings (PLE), to advanced architectures like the encoder-free 12B model, the Mixture-of-Experts (MoE) 26B model, and the highly capable 31B dense model. Each size is optimized for specific use cases, including dedicated vision and audio processing.

Key takeaways

  1. E2B and E4B Models

    These smaller, dense models are optimized for on-device usage and utilize Per-Layer Embeddings (PLE), which are lookup tables for processing queries. They process audio and images using dedicated encoders.

  2. 12B Model Architecture

    This model is suitable for high-end laptops and employs an encoder-free method, removing dedicated encoders (e.g., audio encoder) and directly projecting audio to the LLM.

  3. 26B Model (MoE)

    This Mixture-of-Experts (MoE) model uses 26 billion parameters but only activates four billion at any given time ('A' for active). It features a larger vision encoder, making it ideal for difficult vision tasks.

  4. 31B Model

    This is the most capable dense model in the Gemma 4 family. It utilizes a larger vision encoder and is presented as the top-tier model.

Watch on YouTube Full article

Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind thumbnail

· 21:08

Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind

The talk outlines a framework for multimodal collaborative agents designed to handle 'fuzzy intent' in commerce and consumer verticals. Instead of acting as simple search bar wrappers that assume well-defined user goals, these advanced agents proactively guide users who arrive with only a 'vibe.' The core mechanism is a three-stage loop—Discovery, Research, and Response—which systematically builds a working state from multimodal inputs (images, context) to determine the optimal next question or presentation format.

Key takeaways

  1. Handling Fuzzy Intent 1:48

    Agents must address the 'articulation gap,' recognizing that users often arrive with vague preferences rather than precise keywords. The agent's role is to proactively elicit and refine these fuzzy intents.

  2. The Collaborative Loop 3:23

    The system operates in a loop: Discovery (building the working state), Research (determining the best way to ask/find information), and Response (adapting the output format).

  3. Prioritizing Information Gain 13:41

    The agent must calculate which unknown variable, when queried, will yield the 'maximal information gain' to move the conversation forward efficiently (e.g., determining room width is critical before recommending furniture).

  4. Multimodal Elicitation 5:20

    For subjective preferences, visual inspiration boards and multimodal inputs are significantly more effective than text-based questioning for establishing a common language between the user and the system.

Watch on YouTube Full article

SOTA Generative Media Panel — Dumitru Erhan, Shane Gu & Nicole Brichtova, Google DeepMind thumbnail

· 56:59

SOTA Generative Media Panel — Dumitru Erhan, Shane Gu & Nicole Brichtova, Google DeepMind

The session provided an overview of SOTA generative media models, highlighting new APIs like NanoBanana 2 Lite and Gemini Omni Flash. Key architectural discussions centered on the limitations of language as a sole intermediate representation for complex sensory data (taste, smell, skin tone). The consensus points toward a future requiring unified 'World Models' that integrate visual, temporal, and symbolic reasoning, moving beyond single-modality generation. Evaluation remains highly dependent on human judgment, making robust testing and field feedback critical.

Key takeaways

  1. New APIs Launched for Developers 0:15

    Google launched NanoBanana 2 Lite (the fastest/cheapest image model in the family) and the Gemini Omni Flash APIs. The Omni Flash API enables video generation and editing, priced similarly to V3 fast, making it accessible for developers [0:15-0:40].

  2. Generative Media Capabilities 2:00

    Models can now take diverse inputs (e.g., a storyboard of images, an audio track) to generate video. Furthermore, natural language processing allows for advanced video editing tasks like adding or removing elements from existing footage [1:20-3:00].

  3. The Limitation of Language as Representation 1:50

    Speakers argued that language is an insufficient intermediate representation for highly sensitive sensory data (e.g., taste, smell, skin tone). This suggests a need for more foundational representations, potentially including code or direct binary/latent space conditioning [1:50-2:30].

  4. Evaluation Challenges and Reward Hacking 0:35

    While human preference often favors AI output (e.g., sharper, more saturated images), this metric is unreliable for optimization. External testers have found 'reward hacking' artifacts, such as the model consistently adding wedding rings to hands [0:35-0:45].

Watch on YouTube Full article

Stanford CS547 HCI Seminar | Spring 2026 | Show It or Tell It? Text, Visualization, and Combination thumbnail

· 56:19

Stanford CS547 HCI Seminar | Spring 2026 | Show It or Tell It? Text, Visualization, and Combination

This seminar explores the complex intersection of text and data visualization, arguing that language is a critical component of effective information design. The discussion moves from established cognitive theories (like Dual Coding Theory) to modern AI architectures, detailing how Multimodal Large Language Models (MLLMs) process combined visual and textual inputs using mechanisms like cross-attention. Key findings suggest that while the optimal balance between text and visuals is highly context-dependent, MLLMs are capable of deeply integrating both modalities.

Key takeaways

  1. Language is a key component of visualization 5:19

    Studies show that titles and labels receive long fixations during encoding and are the most likely elements to be recalled, suggesting language significantly impacts how data visualizations are understood. The speaker notes this was historically under-explored in the visualization community.

  2. Optimal design favors annotation over minimalism 21:30

    Research suggests that 'more text is better' for general information displays, provided the text is relevant and properly annotated. This finding challenges traditional minimalist principles in UI/UX design.

  3. MLLMs integrate modalities via cross-attention 41:20

    Multimodal LLMs (MLLMs) process text and visuals by transforming inputs into embeddings. The 'cross-modality embedding architecture' uses a cross-attention mechanism, allowing information to flow between the two distinct sequences (e.g., image tokens interacting with text tokens).

  4. Cognitive processing is context-dependent 30:00

    The speaker notes that understanding how humans integrate text and visuals is complex, citing conflicting evidence across theories (Dual Coding vs. Cognitive Load Theory). The choice of representation depends heavily on the specific task or cognitive ability.

Watch on YouTube Full article

Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber thumbnail

· 21:39

Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber

Uber detailed its approach to building closed-loop evaluation systems for a multimodal agent designed to enhance food photography on Uber Eats. The system addresses the challenge of improving image quality at massive scale (90 billion run rate/year) while preserving merchant authenticity and avoiding homogenization. The architecture involves an end-to-end pipeline incorporating Image Understanding/Routing agents, iterative Image Editing agents, multi-dimensional QA gates, and sophisticated continuous learning loops that detect drift and auto-tune the system without human intervention.

Key takeaways

  1. Closed-Loop System Design 17:07

    The system operates on a closed feedback loop: production data is sampled, compared against human labels (golden source of truth), mismatches are diagnosed by an 'Umbrella Diagnosis Agent,' and the auto-tuning pipeline adjusts agent configurations. This process ensures continuous improvement without requiring constant human intervention.

  2. Handling Model Drift 20:03

    To prevent static models from failing in a live environment, the system uses production data sampling to detect drift. The 'Diagnoser' agent localizes the issue and triggers an auto-tuning pipeline that benchmarks against the golden dataset before deployment.

  3. Evaluation Metrics for Agents 21:05

    For routing agents, evaluation involves measuring precision and recall using a confusion matrix (potentially an N x N matrix if multiple branching paths exist). For the enhancement process, 'Pass at K' measures the pass rate after K iterations of refinement.

Watch on YouTube Full article

Video Has No Memory. Here's How We Built One. — James Le, TwelveLabs thumbnail

· 20:27

Video Has No Memory. Here's How We Built One. — James Le, TwelveLabs

Video AI systems currently lack 'memory,' meaning they process each query from scratch rather than maintaining a durable understanding of an entire video corpus. The solution proposed is building a dedicated memory layer by treating video not as a bag of frames, but as a spatial temporal volume. This architecture utilizes a Context Graph—a navigable structure connecting entities, appearances, relationships, and metadata across time—to enable deep reasoning, advanced tracking, and multi-source analysis (e.g., analyzing an entire World Cup corpus).

Key takeaways

  1. Video is a Spatial Temporal Volume 0:50

    Instead of treating video as a stack of images or text tokens, it must be modeled as a spatial temporal volume that preserves continuity across space, time, visual information, speech, sound, and motion. (0:50)

  2. The Need for a Memory Layer 2:06

    Existing systems suffer from wrong context, weak memory (lacking durable continuity), and insufficient reasoning over complex video data. A dedicated memory layer is required to connect moments across different files or seasons. (2:06)

  3. Context Graph Structure 7:48

    The optimal mental model for video intelligence is a Context Graph, which provides a durable, queryable representation linking time-bounded moments (evidence units), entity appearances, relationships, and corpus-level themes. (7:48)

  4. Core Design Principles 9:04

    To build effective video memory, systems must 'ingest once and reason many times,' store primitives (not just answers), ground every claim to a specific timestamp, let intent shape the memory, and keep the layer composable via APIs. (9:04)

Watch on YouTube Full article

Access all Gemini models with the Interactions API thumbnail

· 1:24

Access all Gemini models with the Interactions API

The Gemini Interactions API is now generally available, providing a unified, single-interface gateway for accessing all Gemini models and agents. This stateful, agent-first ecosystem allows developers to build complex AI applications with persistent memory by passing previous interaction IDs, supporting multimodal inputs like voice notes and images.

Key takeaways

  1. Interactions API General Availability

    The Interactions API is now generally available for use, simplifying access to Gemini models.

  2. Unified Model Access

    It serves as a single interface gateway for all Gemini models and agents, eliminating complex AI workflow setups.

  3. Stateful Memory Support

    The API is stateful, allowing applications to build upon previous interactions by passing the prior interaction ID.

Watch on YouTube Full article

Building an AI-Powered Personal Companion Mobile App - Gerald Versluis - NDC Copenhagen 2026 thumbnail

· 58:54

Building an AI-Powered Personal Companion Mobile App - Gerald Versluis - NDC Copenhagen 2026

The talk demonstrates how to integrate advanced AI functionalities into cross-platform mobile applications using .NET MAUI. The session emphasizes building robust, context-aware companion apps by leveraging abstraction layers and standardized APIs (like the Essentials API). Key architectural patterns include supporting multimodal input (voice, image), utilizing cloud services like Azure AI Foundry for LLMs, and implementing local model support via Onyx for sensitive data scenarios.

Key takeaways

  1. AI Integration is Cross-Platform 17:22

    The implementation of AI features can be abstracted across platforms (iOS, Android, Windows) using .NET MAUI's abstraction layer, ensuring a consistent user experience while allowing platform-specific native behavior.

  2. Context Awareness and Memory 23:50

    AI applications can be significantly enhanced by incorporating context from various sources—including location (geolocation), calendar data, biometrics, and conversation history (memory)—to provide highly personalized user experiences.

  3. Multimodal Input Handling 30:05

    The system supports multiple input types beyond text, including voice recording and image analysis. This requires transforming raw data (audio/image) into a format consumable by Large Language Models (LLMs), such as using speech-to-text transcription.

  4. Model Agnosticism via Interfaces 40:05

    By relying on interfaces (e.g., `I ChatClient`), developers can easily swap out concrete AI implementations—such as those from Microsoft Foundry, OpenAI, or local models like Apple Intelligence/Onyx—without changing core application logic.

Watch on YouTube Full article