# The State of Model Routing — NVIDIA, Cognition, OpenRouter

## Executive summary

The discussion explores the critical field of model routing in a multi-modal AI landscape, emphasizing that simply sending tasks to the best-benchmarked model is fragile. Solutions involve sophisticated orchestration systems (like Cognition's Fusion) that use cheaper models for implementation while leveraging expensive frontier models for high-level planning and decision-making. Key technical challenges include managing context across multiple agents, minimizing costs through KV cache efficiency (e.g., using sidekick agents), and ensuring model reliability when tasks move from in-distribution to out-of-distribution domains.

## Key takeaways

- Model Routing is an Orchestration Problem: Effective AI systems require more than just selecting a single best model; they need robust orchestration that can handle the complexity and changing nature of tasks (e.g., starting as a question, becoming a feature request, then live debugging).
- Cost-Efficiency through Delegation: Advanced routing allows expensive frontier models to handle planning and decision-making, while cheaper mini-models execute the bulk of the work. This approach can significantly reduce costs (e.g., Cognition claims a 40% cost reduction for Fable-level intelligence).
- Context Management is Crucial for Cost Control: Using sidekick agents with continuous running context (keeping the KV cache warm) is more cost-effective than traditional main agent/sub-agent systems, as it drastically reduces costs associated with cached tokens.
- The Danger of Naive Routing: Relying solely on task type for routing is fragile. The complexity and nature of a task change over time, requiring the system to maintain frontier intelligence presence even when delegating work.
- Local vs. Cloud Inference Economics: Self-hosting models offers greater control over cost dynamics and context management (e.g., setting custom cache lifetimes), fundamentally changing the economics compared to relying solely on API providers.

## Technical details

- Model Routing/Fusion: The process of intelligently selecting and coordinating multiple models (e.g., using Cognition's Fusion) to achieve a desired output, optimizing for both performance and cost.
- Sidekick Agents: A continuous context mechanism where one agent maintains the running state (KV cache), preventing the need for constant context reprovisioning and saving significant costs. This is preferred over traditional main/sub-agent architectures.
- Context Compaction: A technique to summarize long contexts, which is necessary for long-running agents. While useful, the speaker warns that compaction is lossy and model intelligence can fall off a cliff well before advertised token limits (e.g., million tokens).
- FlexRun Technology: NVIDIA's technology allowing distillation of a main model into smaller footprints, enabling the system to activate specific classes of weights or sections depending on the task at hand.
- In-Distribution vs. Out-of-Distribution: Small models are highly efficient and effective when tasks remain within their training domain (in-distribution). However, if pushed outside this domain, they can 'thrash' by calling tools in loops, potentially costing more than an expensive frontier model.

## Practical implications

- When designing agentic workflows, prioritize orchestration and state management (harness design) as much as model selection. The system's ability to adapt is more valuable than the single best-performing model.
- For cost optimization, implement sidekick or continuous context mechanisms rather than relying on simple main/sub-agent delegation to maintain cache efficiency.
- When deploying models, rigorously test for out-of-distribution failure modes. A cheap model failing catastrophically is often more expensive than using a robust frontier model.
- Consider the hybrid local/cloud deployment model to maximize compute utilization and gain control over context caching economics.

## Topics

Model Routing, AI Architecture, LLM Economics, Agentic Systems, Context Management (KV Cache), Neotron models, Cognition's Fusion, Devon AI, OpenRouter's Auto Router

Source: https://www.youtube.com/watch?v=QHBjufYK8TA