The Video Signal

Topic

Model Routing

All digests tagged Model Routing

Ask the Experts: How NeMo Switchyard Helps Agents Select Models | Nemotron Labs thumbnail

· 49:23

Ask the Experts: How NeMo Switchyard Helps Agents Select Models | Nemotron Labs

NeMo Switchyard is an open-source model routing library designed for AI agents to solve the problem of relying on a single monolithic LLM. It automatically routes each agent query or step to the optimal model—selecting from any combination of local/cloud and open/closed models—based on real-time needs, optimizing for accuracy, cost, and latency. The system operates beyond simple request routing by tracking state across multi-turn agentic workflows, making it a critical component for building robust, efficient AI systems.

Key takeaways

  1. System of Models Approach 2:49

    The industry is moving away from the 'one model to rule them all' concept toward a 'system of models,' where multiple specialized models are used for different tasks, improving efficiency and capability (0:02:45).

  2. Agent-Aware Routing vs. Simple Routing 5:18

    Switchyard is more than a simple router; it operates on an agentic workflow, tracking state (e.g., tool calls, message history) across multi-turn sessions to make intelligent model selection decisions (0:04:25).

  3. Optimization and Learning 23:18

    The system treats model selection as an optimization problem. It can learn by analyzing agent traces and behavior, predicting potential errors or resource needs to route proactively and save tokens/time (0:21:58).

  4. Full-Stack Routing Flywheel 6:10

    The roadmap envisions a full 'flywheel' of routing, connecting model selection to inference optimization (via NVIDIA Dynamo) and data privacy/anonymization. This allows for continuous improvement across the entire agent lifecycle (0:06:10).

Watch on YouTube Full article

Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOcean thumbnail

· 15:54

Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOcean

The talk argues that relying on public benchmarks to select a single best LLM is flawed; model selection must be based on specific operational preferences. The solution presented is an Inference Router—a purpose-built Mixture of Experts (MoE) model that dynamically selects the optimal LLM for each request based on declared constraints, including task type, cost tolerance, required latency, and system prompts. This approach significantly reduces inference costs and improves performance stability compared to using a single premium model across all tasks.

Key takeaways

  1. Model selection must be preference-driven, not benchmark-driven 4:02

    The 'right' model depends on the specific request context (task, cost, latency) and cannot be determined by a single public leaderboard. For instance, classification may use a small open model for cost efficiency, while code review requires a frontier model for accuracy.

  2. Model orchestration is the new phase of cloud optimization

    Due to exploding inference costs and single-model risk, advanced model orchestration (like routing) is becoming a critical discipline for production applications.

  3. The router optimizes cost and speed per task

    In live demos, the router successfully matched tasks (e.g., code generation, test writing) to specialized models (e.g., GLM 5.2, Claude 3 Sonnet), resulting in significantly lower total session costs (e.g., 8 cents vs. 25 cents) while maintaining comparable quality.

  4. The router is customizable and open-source 5:20

    The solution runs on an open proxy plan and a purpose-built routing model (both open source), allowing users to define custom decision tree rules, set failover policies, and evaluate performance using internal metrics rather than external leaderboards.

Watch on YouTube Full article

The State of Model Routing — NVIDIA, Cognition, OpenRouter thumbnail

· 48:17

The State of Model Routing — NVIDIA, Cognition, OpenRouter

The discussion explores the critical field of model routing in a multi-modal AI landscape, emphasizing that simply sending tasks to the best-benchmarked model is fragile. Solutions involve sophisticated orchestration systems (like Cognition's Fusion) that use cheaper models for implementation while leveraging expensive frontier models for high-level planning and decision-making. Key technical challenges include managing context across multiple agents, minimizing costs through KV cache efficiency (e.g., using sidekick agents), and ensuring model reliability when tasks move from in-distribution to out-of-distribution domains.

Key takeaways

  1. Model Routing is an Orchestration Problem

    Effective AI systems require more than just selecting a single best model; they need robust orchestration that can handle the complexity and changing nature of tasks (e.g., starting as a question, becoming a feature request, then live debugging).

  2. Cost-Efficiency through Delegation 3:35

    Advanced routing allows expensive frontier models to handle planning and decision-making, while cheaper mini-models execute the bulk of the work. This approach can significantly reduce costs (e.g., Cognition claims a 40% cost reduction for Fable-level intelligence).

  3. Context Management is Crucial for Cost Control 6:30

    Using sidekick agents with continuous running context (keeping the KV cache warm) is more cost-effective than traditional main agent/sub-agent systems, as it drastically reduces costs associated with cached tokens.

  4. The Danger of Naive Routing 7:30

    Relying solely on task type for routing is fragile. The complexity and nature of a task change over time, requiring the system to maintain frontier intelligence presence even when delegating work.

  5. Local vs. Cloud Inference Economics 21:45

    Self-hosting models offers greater control over cost dynamics and context management (e.g., setting custom cache lifetimes), fundamentally changing the economics compared to relying solely on API providers.

Watch on YouTube Full article