How MiniMax M3 Was Built: Sparse Attention and Native Multimodality — Olive Song
Olive Song details the architecture and training methodology of MiniMax M3, an open-weight model featuring frontier-level agentic coding, a 1 million token context window, and native multimodality. Key technical innovations include MiniMax Sparse Attention (MSA), which achieves significant speedups (9x faster pre-fill, 15x faster decode) while maintaining compatibility with Grouped-Query Attention (GQA). The talk emphasizes that training the model with text and vision from 'step zero' is superior to post-training additions, and discusses using 3D attention within the Vision Transformer (ViT) for enhanced visual understanding.
Key takeaways
-
MiniMax M3's Frontier Capabilities
2:00
MiniMax M3 is noted as the first open-weight model to combine three frontier capabilities: agentic coding/agent tasks, a 1 million token context window, and native multimodality.
-
MSA Sparse Attention Design
4:17
MSA uses an index branch (lightweight full attention) to identify important tokens, which the sparse branch then attends to. This design is optimized for GQA and addresses infrastructure issues related to GPU memory access and top-k computation dependencies.
-
Native Multimodality Training
15:01
Training with text and vision from 'step zero' (native multimodal) is superior to adding vision later (CPT-style) because it prevents interference with core text capabilities and leads to better integration of visual and textual information, as seen in attention maps.