Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOcean
Summary
The talk argues that relying on public benchmarks to select a single best LLM is flawed; model selection must be based on specific operational preferences. The solution presented is an Inference Router—a purpose-built Mixture of Experts (MoE) model that dynamically selects the optimal LLM for each request based on declared constraints, including task type, cost tolerance, required latency, and system prompts. This approach significantly reduces inference costs and improves performance stability compared to using a single premium model across all tasks.
Key takeaways
-
Model selection must be preference-driven, not benchmark-driven
4:02
The 'right' model depends on the specific request context (task, cost, latency) and cannot be determined by a single public leaderboard. For instance, classification may use a small open model for cost efficiency, while code review requires a frontier model for accuracy.
-
Model orchestration is the new phase of cloud optimization
Due to exploding inference costs and single-model risk, advanced model orchestration (like routing) is becoming a critical discipline for production applications.
-
The router optimizes cost and speed per task
In live demos, the router successfully matched tasks (e.g., code generation, test writing) to specialized models (e.g., GLM 5.2, Claude 3 Sonnet), resulting in significantly lower total session costs (e.g., 8 cents vs. 25 cents) while maintaining comparable quality.
-
The router is customizable and open-source
5:20
The solution runs on an open proxy plan and a purpose-built routing model (both open source), allowing users to define custom decision tree rules, set failover policies, and evaluate performance using internal metrics rather than external leaderboards.
Technical details
-
Inference Router Architecture
The router uses a purpose-built Mixture of Experts (MoE) model that operates in under 200 milliseconds and costs nothing extra. It accepts preferences defined by the user, such as cost constraints, latency requirements, or hard rules.
-
Model Selection Policies
380s
Users can configure multiple models per task (e.g., in 'bug fixing' or 'code generation'). Policies include: manual ranking (forcing a primary model with failover), and 'fastest' selection, which chooses the model that has performed fastest within a defined time window.
-
Evaluation Loop
421s
The system emphasizes an iterative improvement loop: Route -> Evaluate -> Adjust. Users can validate router performance using custom evaluations, comparing metrics like tokens used and speed against single-model benchmarks.
Mentioned resources
- Inference Router
- DigitalOcean Cloud Console
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.