Large clusters for small models — Daniel Svonava, Superlinked
Summary
The talk addresses the shift toward using small, specialized open-source models for AI workloads, which offers massive cost and latency advantages over managed, large-scale APIs. However, serving a diverse fleet of these small models (e.g., a contract review agent using nine different models) presents significant infrastructure challenges. Traditional top-down routing methods and proprietary serving tools are insufficient. The speaker introduces Superlinked's open-source architecture, which utilizes a gateway and a shared queue (using uNATS Jetstream) to allow workers to pull tasks and form their own optimal batches, thereby achieving double the cluster throughput.
Key takeaways
-
Small Models vs. Large APIs
Small open-source models can run on older, more affordable GPUs (e.g., two or three generations old), offering orders of magnitude cost savings and latency improvements compared to managed endpoints.
-
Task Specialization
5:42
Instead of using one large, generalized model, optimal performance is achieved by slicing the workload into tasks and deploying a specific, fine-tuned model for each task (e.g., a contract review agent running nine different models).
-
Infrastructure Bottlenecks
10:37
Serving diverse small models is complicated by three issues: 1) Open-source serving tools require extensive, custom tuning (a 'do-it-yourself' research project). 2) Traditional top-down routers choke on many small, fast requests because their view of worker state is stale. 3) Model adaptation (LoRAs, fine-tunes) creates friction between AI and infrastructure engineers.
-
Decentralized Serving Architecture
17:30
The proposed solution uses a gateway to annotate requests and drop them into a shared queue. Workers then pull tasks from this centralized queue and form their own batches, which significantly improves GPU utilization and cluster throughput.
Technical details
-
Model Deployment & Efficiency
120s
Small models are defined as those that fit entirely on one GPU, making them easier to serve and more affordable. They are highly effective for specific tasks, often outperforming general-purpose models.
-
Serving Architecture
1050s
The recommended topology features a gateway that uses a binary format (like MessagePack) for high throughput, inserts requests into a shared queue (uNATS Jetstream), and allows workers to pull tasks and form batches autonomously, bypassing the limitations of top-down routing.
-
Optimization Techniques
1250s
The system supports diverse runtimes (PyTorch, Kendle, Sglang) and abstracts complexity using a Rust sidecar. An auto-research loop is used to ensure that new models are shipped with all necessary tuning and configuration.
-
Performance Gains
1350s
By centralizing the queue and allowing workers to self-batch, the cluster throughput was doubled. Furthermore, a LoRA trained for German legal text improved retrieval quality by 18% at a low cost (80 cents).
Mentioned resources
- Superlinked
- Apache 2.0
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.