# Is a Faster Model Actually Faster? Test the Whole Job — Morgan Suo, MiniMax

## Executive summary

The core argument is that a faster model (like M3.1 Flash) is only valuable if the entire workflow—the 'finished job'—gets faster and more reliable. Morgan Suo details how optimizing model components (e.g., sparse attention, 4-bit KV cache, speculative decoding) must be evaluated against the final output quality, requiring checks on source verification, calculation accuracy, and the total cost (time and compute) of producing a usable report.

## Key takeaways

- Evaluating Model Speed vs. Job Quality: A model generating a report quickly but missing a critical source or having incorrect calculations does not result in a faster job. The evaluation must focus on the time and cost required for an analyst to verify the report's claims and sources.
- Comprehensive Job Costing: When measuring the cost of a finished job, one must include not only the model computation but also the time and cost associated with retrieval, tool calls, retries, and any manual work needed to fix the report before it is usable.
- Importance of P95 Wait Time: While increased throughput (e.g., 300+ requests/second) is good, the P95 wait time (the time by which 95% of requests arrive) is critical. A significant rise in P95 wait time can negatively impact user experience, even if the median time remains acceptable.

## Technical details

- M3.1 Flash Optimizations: M3.1 Flash incorporates several optimizations: sparse attention on long inputs, 4-bit KV cache (reducing stored/moved data), low-precision experts (using 4-bit weights/activations), speculative decoding, and reasoning budgets. Each change must be checked against the desired output quality.
- Sparse Attention: Sparse attention uses a selection step to score blocks of text, calculating attention only on selected blocks while keeping nearby text in view. This reduces attention work across long documents while maintaining the ability to find critical evidence (e.g., a footnote).
- 4-bit KV Cache: The model reduces the storage and movement of internal information (Keys and Values) by changing them from 8 bits to 4 bits, while the query remains at 8 bits. This reduces data movement and storage requirements.
- Speculative Decoding: This serving method operates in two stages: a faster draft process proposes a short sequence of tokens, and the target model tracks it, accepting the tokens and making corrections as needed. The total time must account for drafting, checking, and rejecting proposed tokens.
- Reasoning Budgets: Reasoning budgets allow controlling the amount of work the model performs (e.g., setting a 'thinking budget'). The trade-off is clear: higher reasoning effort (max) may be necessary for accuracy, even if it uses more output tokens.

## Practical implications

- When integrating a new LLM, define a rigorous 'finished job' metric that includes source verification, calculation checks, and the total cost (time, compute, and manual effort) of the entire workflow.
- Do not rely solely on throughput or median latency; always measure P95 wait time to accurately gauge user experience.
- When optimizing, treat the model's output as a component of a larger system. The optimization must demonstrably improve the final, usable report.

## Topics

Large Language Models, Model Optimization, Workflow Engineering, AI Evaluation Metrics, Compute Efficiency, Morgan Suo, MiniMax, AIE CODE summit

Source: https://www.youtube.com/watch?v=YvQ4BVTWfHI