Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax
The discussion details MiniMax M3, a frontier open-source model featuring a functional one-million-token context window and native multimodality (text, image, video). The efficiency of this long context is achieved through the proprietary Minimax Sparse Attention (MSA) architecture. Furthermore, the conversation highlights advanced research practices at MiniMax, including using intern contributions for architectural improvements and training models from the very first step to ensure optimal multimodal performance.
Key takeaways
-
MiniMax M3 Capabilities
4:53
M3 is a highly capable open-source model (400B total parameters, 20B activated) that combines coding proficiency, agentic capabilities, and multimodal understanding across text, images, and video. It features a functional one-million-token context window.
-
Long Context Necessity for Agents
6:30
The need for extremely long context windows is driven by complex agentic tasks, which require processing multi-round conversations and accumulating tool responses from an entire environment, making shorter contexts insufficient.
-
Sparse Attention Architecture (MSA)
7:30
The Minimax Sparse Attention (MSA) architecture is designed to efficiently scale context length. It uses an index branch to select relevant context blocks, followed by a sparse attention branch for calculation, allowing scalability in both length and model size.
-
Native Multimodality Training
17:25
MiniMax trains M3 from the very first step as a multimodal model (native multimodality), rather than adding vision capabilities after text pre-training. This approach prevents performance degradation in text understanding and ensures better scalability.
-
Internal Research Culture
13:49
MiniMax utilizes an open research culture where external community members (including interns) can propose projects, find model weaknesses through evaluations, and contribute code/improvements that are integrated into final training runs.