Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax

Key takeaways

1. MiniMax M3 Capabilities 4:53

   M3 is a highly capable open-source model (400B total parameters, 20B activated) that combines coding proficiency, agentic capabilities, and multimodal understanding across text, images, and video. It features a functional one-million-token context window.

2. Long Context Necessity for Agents 6:30

   The need for extremely long context windows is driven by complex agentic tasks, which require processing multi-round conversations and accumulating tool responses from an entire environment, making shorter contexts insufficient.

3. Sparse Attention Architecture (MSA) 7:30

   The Minimax Sparse Attention (MSA) architecture is designed to efficiently scale context length. It uses an index branch to select relevant context blocks, followed by a sparse attention branch for calculation, allowing scalability in both length and model size.

4. Native Multimodality Training 17:25

   MiniMax trains M3 from the very first step as a multimodal model (native multimodality), rather than adding vision capabilities after text pre-training. This approach prevents performance degradation in text understanding and ensures better scalability.

5. Internal Research Culture 13:49

   MiniMax utilizes an open research culture where external community members (including interns) can propose projects, find model weaknesses through evaluations, and contribute code/improvements that are integrated into final training runs.