The Video Signal

Topic

Multimodality

All digests tagged Multimodality

Speech-to-Speech Model Research at Google DeepMind — Valeria Wu Fon & Tom Ouyang, Google DeepMind thumbnail

· 16:42

Speech-to-Speech Model Research at Google DeepMind — Valeria Wu Fon & Tom Ouyang, Google DeepMind

Google DeepMind presented research on Speech-to-Speech models, positioning them as the foundation for the 'agentic future' of voice interaction. The core argument is that modern models must move beyond simple Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) chains. By leveraging natively multimodal pre-training (audio, video, text), these models achieve a 'trifecta' of conversational fluency, high intelligence (task completion/reasoning), and multimodality (handling video, screen shares, and documents). This enables complex, real-time applications like live multilingual translation and proactive, low-latency conversational agents.

Key takeaways

  1. The Shift from Cascaded to Unified Models 2:31

    Historically, speech processing required multiple hand-built components (feature extraction, acoustic modeling, language modeling, rescoring). Modern LLMs, trained on interleaved multimodal examples, collapse this chain, allowing a single model to understand and transition between audio, video, and text inputs.

  2. The Three Pillars of Speech-to-Speech Models 10:01

    A robust model must balance three vectors: 1) Conversational (low latency/snappy); 2) Intelligent (task completion, instruction following); and 3) Multimodal (accepting video, screen shares, and documents). Improving one vector often degrades the others (e.g., increasing intelligence can decrease time to first audio).

  3. Real-Time Multilingual Translation 5:27

    The model can perform streaming, real-time translation across multiple speakers and languages (e.g., English, Spanish, Italian, Chinese) with quality comparable to offline systems, a capability difficult for cascaded systems.

  4. Proactive Audio and Multimodal Output 13:57

    Advanced features include 'proactive audio,' where the model knows when to respond despite background noise or when another speaker is talking. Furthermore, the model can generate multimodal output, including customized real-time avatars with low-latency lip-syncing.

Watch on YouTube Full article

Celebrating one billion Gemma downloads thumbnail

· 0:56

Celebrating one billion Gemma downloads

Google Developers celebrated reaching one billion downloads for the Gemma model family. The discussion highlighted significant advancements in multimodal AI capabilities, specifically noting that Gemma 4 supports video and audio understanding. For developers, key takeaways include utilizing Unsloth Desktop—a local coding agent—for development and fine-tuning, and keeping an eye on future platforms like GenieX, which is designed to integrate highly requested models like Gemma.

Key takeaways

  1. Gemma's Multimodal Capabilities

    The latest model in the family, Gemma 4, represents a major breakthrough by supporting both video and audio understanding.

  2. Local Development Tools

    Unsloth Desktop was launched as a local coding agent, allowing developers to run models and perform tasks entirely offline. Users can also fine-tune models locally.

  3. Future Platform Roadmap

    A platform called GenieX is under development, positioning Gemma as one of the most requested models for future enterprise integration. The team expressed excitement for upcoming versions, including Gemma 5 and Gemma 6.

Watch on YouTube Full article

Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax thumbnail

· 20:48

Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax

The discussion details MiniMax M3, a frontier open-source model featuring a functional one-million-token context window and native multimodality (text, image, video). The efficiency of this long context is achieved through the proprietary Minimax Sparse Attention (MSA) architecture. Furthermore, the conversation highlights advanced research practices at MiniMax, including using intern contributions for architectural improvements and training models from the very first step to ensure optimal multimodal performance.

Key takeaways

  1. MiniMax M3 Capabilities 4:53

    M3 is a highly capable open-source model (400B total parameters, 20B activated) that combines coding proficiency, agentic capabilities, and multimodal understanding across text, images, and video. It features a functional one-million-token context window.

  2. Long Context Necessity for Agents 6:30

    The need for extremely long context windows is driven by complex agentic tasks, which require processing multi-round conversations and accumulating tool responses from an entire environment, making shorter contexts insufficient.

  3. Sparse Attention Architecture (MSA) 7:30

    The Minimax Sparse Attention (MSA) architecture is designed to efficiently scale context length. It uses an index branch to select relevant context blocks, followed by a sparse attention branch for calculation, allowing scalability in both length and model size.

  4. Native Multimodality Training 17:25

    MiniMax trains M3 from the very first step as a multimodal model (native multimodality), rather than adding vision capabilities after text pre-training. This approach prevents performance degradation in text understanding and ensures better scalability.

  5. Internal Research Culture 13:49

    MiniMax utilizes an open research culture where external community members (including interns) can propose projects, find model weaknesses through evaluations, and contribute code/improvements that are integrated into final training runs.

Watch on YouTube Full article

Bringing agents onto the world wide web — Paul Klein IV, Browserbase thumbnail

· 18:26

Bringing agents onto the world wide web — Paul Klein IV, Browserbase

The talk argues that the current bottleneck for building reliable web agents is not model capability (AI), but rather the missing engineering infrastructure—the 'harness' and scalable systems. Reliable browser agents must be multimodal, harness-engineered with memory/skills, and operate on consistent, robust infrastructure. The speaker emphasizes that solving this overhang is an engineering problem that companies can tackle today, accelerating AI diffusion into real-world enterprise use cases (e.g., logistics, banking).

Key takeaways

  1. The bottleneck is the harness, not the model. 7:12

    Model capabilities have advanced significantly; therefore, agents struggle due to missing scaffolding and systems around the LLM (the 'harness'). Building a custom harness optimized for a domain can improve results beyond baseline models. (4:32)

  2. Reliable browser agents require three components. 11:45

    They must be multimodal, harness-engineered (using skills and memory), and run on reliable infrastructure that guarantees consistent page rendering across all runs. (7:05)

  3. The web needs to adapt for agents. 17:04

    Key improvements required from the web include better accessibility trees, adoption of Chrome's Web MCP, solving agent login/authentication securely, and establishing a trust framework (a 'Verisign moment') for agents. (10:24)

  4. Infrastructure must be scalable and consistent. 15:09

    Production environments require infrastructure that can handle thousands of agents, maintain consistency (e.g., preventing mobile/desktop layout shifts), and ideally achieve SOC 2 compliance at scale. (9:09)

Watch on YouTube Full article

The New Primitives: Building AI Native Software — Kwindla Kramer, Daily thumbnail

· 21:14

The New Primitives: Building AI Native Software — Kwindla Kramer, Daily

The talk traces the 80-year history of digital computing—from Vannevar Bush's predictions in 1945 to modern AI agents—to argue that current 'agents' are merely a primitive. The speaker posits that just as web pages were superseded by full web and mobile applications, agents will eventually give way to a new fully AI native software layer. This next generation requires advanced primitives like asynchronous non-blocking context compression and dynamic interface generation.

Key takeaways

  1. The Evolution of Primitives 6:52

    History shows that every major computing leap (e.g., web pages to mobile apps) renders the previous primitive insufficient for the next era. Agents are viewed as the 'web page' of the current AI age.

  2. The Next Frontier: AI Native Software 20:05

    Building beyond agents requires mastering primitives such as asynchronous non-blocking context compression, long running subagents that share context, progressive skills loading, dynamic interface generation, and conversational voice.

  3. The Role of Abstraction 17:15

    Historical examples like VisiCalc demonstrate how new abstractions (e.g., the spreadsheet) make vastly more complex work possible, creating entirely new categories of work rather than eliminating jobs.

Watch on YouTube Full article

Agents at Scale: Inside MiniMax's Model and the Infrastructure Behind It — Olive Song thumbnail

· 20:14

Agents at Scale: Inside MiniMax's Model and the Infrastructure Behind It — Olive Song

The discussion details the engineering stack and open-source philosophy behind MiniMax's model, M3. Olive Song emphasizes that the open weights approach allows the community to build upon and optimize the model, fostering widespread intelligence access. The technical focus covers advanced training techniques—including multimodality (text, image, video) and Reinforcement Learning (RL) for long-horizon tasks like replicating academic papers (12-hour runs)—and the complex infrastructure required for deployment. Key engineering challenges discussed include writing specialized GPU kernels, optimizing the inference stack from 'day zero,' managing KV cache growth in agentic workflows, and adapting to shifting workloads from chat-based to multi-turn, tool-calling agents.

Key takeaways

  1. Open Weights Philosophy 2:07

    MiniMax advocates for open source because it aligns with their mission of making intelligence widely accessible. By releasing weights, they enable developers (like Together AI) to optimize the model's inference speed and capabilities through community contributions.

  2. Multimodality Training 8:02

    MiniMax M3 is multimodal, understanding text, code, images, and videos. Crucially, it was trained multimodally from scratch to prevent 'training collapse,' ensuring that the modalities naturally interact (e.g., visual tokens attending to text tokens).

  3. Agentic Workloads and Inference Shifts 13:40

    The workload is shifting from simple chat turns to complex agentic workflows involving hundreds of multi-turn tool calls. This requires significant optimization in the inference stack, particularly concerning KV cache management and routing.

  4. Long-Horizon RL Tasks

    Training for complex tasks (e.g., replicating an ICLR paper over 12 hours) requires careful formulation of the problem, defining environments, and optimizing reward functions within the Reinforcement Learning framework.

Watch on YouTube Full article

New Model: Inkling by Thinking Machine on Hugging Face thumbnail

· 36:00

New Model: Inkling by Thinking Machine on Hugging Face

Thinking Machines announced Inkling, a massive open model with nearly one trillion parameters. It is designed as a natively multimodal architecture, accepting image, text, and audio inputs simultaneously. Key technical features include a Mixture-of-Experts (MoE) structure (975B total / 41B active params), a 1M token context window, and advanced deployment support across multiple frameworks like `transformers`, SGLang, vLLM, and `llama.cpp`. The model is available in BF16 and NVFP4 formats, facilitating high-performance inference on various hardware setups.

Key takeaways

  1. Multimodal Capability & Scale 3:50

    Inkling is a true multimodal model that natively processes image, text, and audio inputs using a single architecture. It boasts an immense 1M token context window and was trained on 45T tokens.

  2. Architectural Innovations 5:10

    The model utilizes relative attention (replacing RoPE) and a specialized SConv layer to efficiently aggregate hidden states, making it highly efficient for multimodal tasks compared to previous models.

  3. Deployment Flexibility 7:10

    Inkling supports multiple deployment paths: BF16 (requiring ~2TB VRAM) and NVFP4 (600GB VRAM). It provides day-zero support for `transformers`, SGLang, vLLM, and `llama.cpp` (including GGUF quants via Unsloth), enabling diverse inference environments.

Watch on YouTube Full article