Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOcean
The talk argues that relying on public benchmarks to select a single best LLM is flawed; model selection must be based on specific operational preferences. The solution presented is an Inference Router—a purpose-built Mixture of Experts (MoE) model that dynamically selects the optimal LLM for each request based on declared constraints, including task type, cost tolerance, required latency, and system prompts. This approach significantly reduces inference costs and improves performance stability compared to using a single premium model across all tasks.
Key takeaways
-
Model selection must be preference-driven, not benchmark-driven
4:02
The 'right' model depends on the specific request context (task, cost, latency) and cannot be determined by a single public leaderboard. For instance, classification may use a small open model for cost efficiency, while code review requires a frontier model for accuracy.
-
Model orchestration is the new phase of cloud optimization
Due to exploding inference costs and single-model risk, advanced model orchestration (like routing) is becoming a critical discipline for production applications.
-
The router optimizes cost and speed per task
In live demos, the router successfully matched tasks (e.g., code generation, test writing) to specialized models (e.g., GLM 5.2, Claude 3 Sonnet), resulting in significantly lower total session costs (e.g., 8 cents vs. 25 cents) while maintaining comparable quality.
-
The router is customizable and open-source
5:20
The solution runs on an open proxy plan and a purpose-built routing model (both open source), allowing users to define custom decision tree rules, set failover policies, and evaluate performance using internal metrics rather than external leaderboards.