# Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax

## Executive summary

The discussion details MiniMax M3, a frontier open-source model featuring a functional one-million-token context window and native multimodality (text, image, video). The efficiency of this long context is achieved through the proprietary Minimax Sparse Attention (MSA) architecture. Furthermore, the conversation highlights advanced research practices at MiniMax, including using intern contributions for architectural improvements and training models from the very first step to ensure optimal multimodal performance.

## Key takeaways

- MiniMax M3 Capabilities: M3 is a highly capable open-source model (400B total parameters, 20B activated) that combines coding proficiency, agentic capabilities, and multimodal understanding across text, images, and video. It features a functional one-million-token context window.
- Long Context Necessity for Agents: The need for extremely long context windows is driven by complex agentic tasks, which require processing multi-round conversations and accumulating tool responses from an entire environment, making shorter contexts insufficient.
- Sparse Attention Architecture (MSA): The Minimax Sparse Attention (MSA) architecture is designed to efficiently scale context length. It uses an index branch to select relevant context blocks, followed by a sparse attention branch for calculation, allowing scalability in both length and model size.
- Native Multimodality Training: MiniMax trains M3 from the very first step as a multimodal model (native multimodality), rather than adding vision capabilities after text pre-training. This approach prevents performance degradation in text understanding and ensures better scalability.
- Internal Research Culture: MiniMax utilizes an open research culture where external community members (including interns) can propose projects, find model weaknesses through evaluations, and contribute code/improvements that are integrated into final training runs.

## Technical details

- Model Architecture: The Minimax Sparse Attention (MSA) architecture is used to achieve the functional one-million-token context window efficiently. It involves an index branch and a sparse attention branch.
- Multimodality Training: The model employs 'native multimodality,' meaning text, image, and video understanding are trained simultaneously from the initial pre-training step to maintain optimal performance across all modalities.
- Model Specifications: M3 has a total of approximately 400 billion parameters, with 20 billion activated parameters. It is designed to handle text, images, and video inputs.

## Practical implications

- The combination of long context and multimodality enables advanced agent applications, such as having an agent watch a YouTube tutorial or reading unstructured reports (e.g., PowerPoint files) to perform tasks.
- The focus on efficient architecture (like MSA) suggests future models will prioritize resource efficiency alongside capability scaling, potentially reducing inference costs for complex tasks.
- MiniMax's open research model encourages community contribution, suggesting that external feedback is a critical component of frontier model improvement.

## Topics

Large Language Models (LLMs), Multimodality, Context Window Scaling, Agentic AI, Sparse Attention Architectures, MiniMax M3, Hugging Face

Source: https://www.youtube.com/watch?v=5Cxe5dv2Xlw