# Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOcean

## Executive summary

The talk argues that relying on public benchmarks to select a single best LLM is flawed; model selection must be based on specific operational preferences. The solution presented is an Inference Router—a purpose-built Mixture of Experts (MoE) model that dynamically selects the optimal LLM for each request based on declared constraints, including task type, cost tolerance, required latency, and system prompts. This approach significantly reduces inference costs and improves performance stability compared to using a single premium model across all tasks.

## Key takeaways

- Model selection must be preference-driven, not benchmark-driven: The 'right' model depends on the specific request context (task, cost, latency) and cannot be determined by a single public leaderboard. For instance, classification may use a small open model for cost efficiency, while code review requires a frontier model for accuracy.
- Model orchestration is the new phase of cloud optimization: Due to exploding inference costs and single-model risk, advanced model orchestration (like routing) is becoming a critical discipline for production applications.
- The router optimizes cost and speed per task: In live demos, the router successfully matched tasks (e.g., code generation, test writing) to specialized models (e.g., GLM 5.2, Claude 3 Sonnet), resulting in significantly lower total session costs (e.g., 8 cents vs. 25 cents) while maintaining comparable quality.
- The router is customizable and open-source: The solution runs on an open proxy plan and a purpose-built routing model (both open source), allowing users to define custom decision tree rules, set failover policies, and evaluate performance using internal metrics rather than external leaderboards.

## Technical details

- Inference Router Architecture: The router uses a purpose-built Mixture of Experts (MoE) model that operates in under 200 milliseconds and costs nothing extra. It accepts preferences defined by the user, such as cost constraints, latency requirements, or hard rules.
- Model Selection Policies: Users can configure multiple models per task (e.g., in 'bug fixing' or 'code generation'). Policies include: manual ranking (forcing a primary model with failover), and 'fastest' selection, which chooses the model that has performed fastest within a defined time window.
- Evaluation Loop: The system emphasizes an iterative improvement loop: Route -> Evaluate -> Adjust. Users can validate router performance using custom evaluations, comparing metrics like tokens used and speed against single-model benchmarks.

## Practical implications

- Implement model routing layers to mitigate cost overruns and single-point-of-failure risks associated with relying on a single premium LLM.
- Design your application workflow to define explicit constraints (cost, latency) for different tasks rather than assuming one monolithic model can handle everything.
- Utilize the router's evaluation capabilities to prove that dynamic routing provides superior performance and cost efficiency compared to static model selection.

## Topics

LLM Orchestration, Model Routing, Inference Engine, Cost Optimization, Mixture of Experts (MoE), AI Infrastructure, Inference Router, DigitalOcean Cloud Console

Source: https://www.youtube.com/watch?v=FvxY8oPoI8o