The State of Model Routing — NVIDIA, Cognition, OpenRouter
The discussion explores the critical field of model routing in a multi-modal AI landscape, emphasizing that simply sending tasks to the best-benchmarked model is fragile. Solutions involve sophisticated orchestration systems (like Cognition's Fusion) that use cheaper models for implementation while leveraging expensive frontier models for high-level planning and decision-making. Key technical challenges include managing context across multiple agents, minimizing costs through KV cache efficiency (e.g., using sidekick agents), and ensuring model reliability when tasks move from in-distribution to out-of-distribution domains.
Key takeaways
-
Model Routing is an Orchestration Problem
Effective AI systems require more than just selecting a single best model; they need robust orchestration that can handle the complexity and changing nature of tasks (e.g., starting as a question, becoming a feature request, then live debugging).
-
Cost-Efficiency through Delegation
3:35
Advanced routing allows expensive frontier models to handle planning and decision-making, while cheaper mini-models execute the bulk of the work. This approach can significantly reduce costs (e.g., Cognition claims a 40% cost reduction for Fable-level intelligence).
-
Context Management is Crucial for Cost Control
6:30
Using sidekick agents with continuous running context (keeping the KV cache warm) is more cost-effective than traditional main agent/sub-agent systems, as it drastically reduces costs associated with cached tokens.
-
The Danger of Naive Routing
7:30
Relying solely on task type for routing is fragile. The complexity and nature of a task change over time, requiring the system to maintain frontier intelligence presence even when delegating work.
-
Local vs. Cloud Inference Economics
21:45
Self-hosting models offers greater control over cost dynamics and context management (e.g., setting custom cache lifetimes), fundamentally changing the economics compared to relying solely on API providers.