Large clusters for small models — Daniel Svonava, Superlinked
The talk addresses the shift toward using small, specialized open-source models for AI workloads, which offers massive cost and latency advantages over managed, large-scale APIs. However, serving a diverse fleet of these small models (e.g., a contract review agent using nine different models) presents significant infrastructure challenges. Traditional top-down routing methods and proprietary serving tools are insufficient. The speaker introduces Superlinked's open-source architecture, which utilizes a gateway and a shared queue (using uNATS Jetstream) to allow workers to pull tasks and form their own optimal batches, thereby achieving double the cluster throughput.
Key takeaways
-
Small Models vs. Large APIs
Small open-source models can run on older, more affordable GPUs (e.g., two or three generations old), offering orders of magnitude cost savings and latency improvements compared to managed endpoints.
-
Task Specialization
5:42
Instead of using one large, generalized model, optimal performance is achieved by slicing the workload into tasks and deploying a specific, fine-tuned model for each task (e.g., a contract review agent running nine different models).
-
Infrastructure Bottlenecks
10:37
Serving diverse small models is complicated by three issues: 1) Open-source serving tools require extensive, custom tuning (a 'do-it-yourself' research project). 2) Traditional top-down routers choke on many small, fast requests because their view of worker state is stale. 3) Model adaptation (LoRAs, fine-tunes) creates friction between AI and infrastructure engineers.
-
Decentralized Serving Architecture
17:30
The proposed solution uses a gateway to annotate requests and drop them into a shared queue. Workers then pull tasks from this centralized queue and form their own batches, which significantly improves GPU utilization and cluster throughput.