The Video Signal

Topic

MiniMax M3

All digests tagged MiniMax M3

Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax thumbnail

· 20:48

Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax

The discussion details MiniMax M3, a frontier open-source model featuring a functional one-million-token context window and native multimodality (text, image, video). The efficiency of this long context is achieved through the proprietary Minimax Sparse Attention (MSA) architecture. Furthermore, the conversation highlights advanced research practices at MiniMax, including using intern contributions for architectural improvements and training models from the very first step to ensure optimal multimodal performance.

Key takeaways

  1. MiniMax M3 Capabilities 4:53

    M3 is a highly capable open-source model (400B total parameters, 20B activated) that combines coding proficiency, agentic capabilities, and multimodal understanding across text, images, and video. It features a functional one-million-token context window.

  2. Long Context Necessity for Agents 6:30

    The need for extremely long context windows is driven by complex agentic tasks, which require processing multi-round conversations and accumulating tool responses from an entire environment, making shorter contexts insufficient.

  3. Sparse Attention Architecture (MSA) 7:30

    The Minimax Sparse Attention (MSA) architecture is designed to efficiently scale context length. It uses an index branch to select relevant context blocks, followed by a sparse attention branch for calculation, allowing scalability in both length and model size.

  4. Native Multimodality Training 17:25

    MiniMax trains M3 from the very first step as a multimodal model (native multimodality), rather than adding vision capabilities after text pre-training. This approach prevents performance degradation in text understanding and ensures better scalability.

  5. Internal Research Culture 13:49

    MiniMax utilizes an open research culture where external community members (including interns) can propose projects, find model weaknesses through evaluations, and contribute code/improvements that are integrated into final training runs.

Watch on YouTube Full article

Agents at Scale: Inside MiniMax's Model and the Infrastructure Behind It — Olive Song thumbnail

· 20:14

Agents at Scale: Inside MiniMax's Model and the Infrastructure Behind It — Olive Song

The discussion details the engineering stack and open-source philosophy behind MiniMax's model, M3. Olive Song emphasizes that the open weights approach allows the community to build upon and optimize the model, fostering widespread intelligence access. The technical focus covers advanced training techniques—including multimodality (text, image, video) and Reinforcement Learning (RL) for long-horizon tasks like replicating academic papers (12-hour runs)—and the complex infrastructure required for deployment. Key engineering challenges discussed include writing specialized GPU kernels, optimizing the inference stack from 'day zero,' managing KV cache growth in agentic workflows, and adapting to shifting workloads from chat-based to multi-turn, tool-calling agents.

Key takeaways

  1. Open Weights Philosophy 2:07

    MiniMax advocates for open source because it aligns with their mission of making intelligence widely accessible. By releasing weights, they enable developers (like Together AI) to optimize the model's inference speed and capabilities through community contributions.

  2. Multimodality Training 8:02

    MiniMax M3 is multimodal, understanding text, code, images, and videos. Crucially, it was trained multimodally from scratch to prevent 'training collapse,' ensuring that the modalities naturally interact (e.g., visual tokens attending to text tokens).

  3. Agentic Workloads and Inference Shifts 13:40

    The workload is shifting from simple chat turns to complex agentic workflows involving hundreds of multi-turn tool calls. This requires significant optimization in the inference stack, particularly concerning KV cache management and routing.

  4. Long-Horizon RL Tasks

    Training for complex tasks (e.g., replicating an ICLR paper over 12 hours) requires careful formulation of the problem, defining environments, and optimizing reward functions within the Reinforcement Learning framework.

Watch on YouTube Full article