# Large clusters for small models — Daniel Svonava, Superlinked

## Executive summary

The talk addresses the shift toward using small, specialized open-source models for AI workloads, which offers massive cost and latency advantages over managed, large-scale APIs. However, serving a diverse fleet of these small models (e.g., a contract review agent using nine different models) presents significant infrastructure challenges. Traditional top-down routing methods and proprietary serving tools are insufficient. The speaker introduces Superlinked's open-source architecture, which utilizes a gateway and a shared queue (using uNATS Jetstream) to allow workers to pull tasks and form their own optimal batches, thereby achieving double the cluster throughput.

## Key takeaways

- Small Models vs. Large APIs: Small open-source models can run on older, more affordable GPUs (e.g., two or three generations old), offering orders of magnitude cost savings and latency improvements compared to managed endpoints.
- Task Specialization: Instead of using one large, generalized model, optimal performance is achieved by slicing the workload into tasks and deploying a specific, fine-tuned model for each task (e.g., a contract review agent running nine different models).
- Infrastructure Bottlenecks: Serving diverse small models is complicated by three issues: 1) Open-source serving tools require extensive, custom tuning (a 'do-it-yourself' research project). 2) Traditional top-down routers choke on many small, fast requests because their view of worker state is stale. 3) Model adaptation (LoRAs, fine-tunes) creates friction between AI and infrastructure engineers.
- Decentralized Serving Architecture: The proposed solution uses a gateway to annotate requests and drop them into a shared queue. Workers then pull tasks from this centralized queue and form their own batches, which significantly improves GPU utilization and cluster throughput.

## Technical details

- Model Deployment & Efficiency: Small models are defined as those that fit entirely on one GPU, making them easier to serve and more affordable. They are highly effective for specific tasks, often outperforming general-purpose models.
- Serving Architecture: The recommended topology features a gateway that uses a binary format (like MessagePack) for high throughput, inserts requests into a shared queue (uNATS Jetstream), and allows workers to pull tasks and form batches autonomously, bypassing the limitations of top-down routing.
- Optimization Techniques: The system supports diverse runtimes (PyTorch, Kendle, Sglang) and abstracts complexity using a Rust sidecar. An auto-research loop is used to ensure that new models are shipped with all necessary tuning and configuration.
- Performance Gains: By centralizing the queue and allowing workers to self-batch, the cluster throughput was doubled. Furthermore, a LoRA trained for German legal text improved retrieval quality by 18% at a low cost (80 cents).

## Practical implications

- Adopting small, specialized models allows for massive cost and latency improvements over proprietary managed APIs.
- The infrastructure challenge shifts from model availability to efficient, decentralized serving architecture.
- The use of open-source, self-hosted stacks (e.g., Rust sidecar, shared queues) is necessary to manage the complexity of diverse model runtimes and adaptations (LoRAs).

## Topics

Small Language Models, AI Inference, Distributed Systems, Open Source AI, GPU Optimization, LLM Serving, Superlinked, Apache 2.0

Source: https://www.youtube.com/watch?v=g4SsanB0gMc