Ask the Experts: How NeMo Switchyard Helps Agents Select Models | Nemotron Labs
NeMo Switchyard is an open-source model routing library designed for AI agents to solve the problem of relying on a single monolithic LLM. It automatically routes each agent query or step to the optimal model—selecting from any combination of local/cloud and open/closed models—based on real-time needs, optimizing for accuracy, cost, and latency. The system operates beyond simple request routing by tracking state across multi-turn agentic workflows, making it a critical component for building robust, efficient AI systems.
Key takeaways
-
System of Models Approach
2:49
The industry is moving away from the 'one model to rule them all' concept toward a 'system of models,' where multiple specialized models are used for different tasks, improving efficiency and capability (0:02:45).
-
Agent-Aware Routing vs. Simple Routing
5:18
Switchyard is more than a simple router; it operates on an agentic workflow, tracking state (e.g., tool calls, message history) across multi-turn sessions to make intelligent model selection decisions (0:04:25).
-
Optimization and Learning
23:18
The system treats model selection as an optimization problem. It can learn by analyzing agent traces and behavior, predicting potential errors or resource needs to route proactively and save tokens/time (0:21:58).
-
Full-Stack Routing Flywheel
6:10
The roadmap envisions a full 'flywheel' of routing, connecting model selection to inference optimization (via NVIDIA Dynamo) and data privacy/anonymization. This allows for continuous improvement across the entire agent lifecycle (0:06:10).