GPU Optimization — The Video Signal

Sep 19, 2026 · 25:07

Large clusters for small models — Daniel Svonava, Superlinked

The talk addresses the shift toward using small, specialized open-source models for AI workloads, which offers massive cost and latency advantages over managed, large-scale APIs. However, serving a diverse fleet of these small models (e.g., a contract review agent using nine different models) presents significant infrastructure challenges. Traditional top-down routing methods and proprietary serving tools are insufficient. The speaker introduces Superlinked's open-source architecture, which utilizes a gateway and a shared queue (using uNATS Jetstream) to allow workers to pull tasks and form their own optimal batches, thereby achieving double the cluster throughput.

Key takeaways

Small Models vs. Large APIs
Small open-source models can run on older, more affordable GPUs (e.g., two or three generations old), offering orders of magnitude cost savings and latency improvements compared to managed endpoints.
Task Specialization 5:42
Instead of using one large, generalized model, optimal performance is achieved by slicing the workload into tasks and deploying a specific, fine-tuned model for each task (e.g., a contract review agent running nine different models).
Infrastructure Bottlenecks 10:37
Serving diverse small models is complicated by three issues: 1) Open-source serving tools require extensive, custom tuning (a 'do-it-yourself' research project). 2) Traditional top-down routers choke on many small, fast requests because their view of worker state is stale. 3) Model adaptation (LoRAs, fine-tunes) creates friction between AI and infrastructure engineers.
Decentralized Serving Architecture 17:30
The proposed solution uses a gateway to annotate requests and drop them into a shared queue. Workers then pull tasks from this centralized queue and form their own batches, which significantly improves GPU utilization and cluster throughput.

Watch on YouTube Full article

Sep 8, 2026 · 1:28:12

Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher

This workshop provides a deep dive into optimizing Large Language Model (LLM) inference at scale. The core challenge is managing the high computational and memory costs associated with LLMs, particularly as context length and concurrent users increase. Solutions are categorized into two areas: model optimizations (e.g., quantization, Grouped Query Attention, Flash Attention) to reduce size/compute, and serving optimizations (e.g., Paged Attention, Continuous Batching, Prefix Caching) to improve throughput and memory efficiency. The session concludes with a comparative analysis of leading inference engines like vLLM and SGLang.

Key takeaways

Inference Cost Drivers 0:05
The primary pain points are: 1) Memory consumption that grows with context length (due to the KV cache); 2) Time To First Token (TTFT) degradation as input size increases; and 3) Throughput collapse when requests are processed sequentially.
Memory Scaling is Critical 0:10
The Key-Value (KV) cache memory usage scales significantly: for a Mistral 7B model, the KV size per token is approximately 131 KB. Serving multiple users with long contexts quickly exhausts GPU VRAM (e.g., 80 concurrent users and 4K context can require 42 GB).
Serving Optimizations are Essential 0:24
To maximize throughput, techniques like Paged Attention (borrowed from OS paging) solve memory fragmentation; Continuous Batching keeps the GPU occupied by processing multiple requests efficiently; and Prefix Caching saves computation when handling similar prompts across different users.
Engine Selection Trade-offs 0:46
While vLLM and SGLang show no statistical difference on standard workloads, SGLang demonstrated a significant advantage (three to four times better) when handling complex agentic branching workflows.

Watch on YouTube Full article

Jul 31, 2026 · 20:14

Agents at Scale: Inside MiniMax's Model and the Infrastructure Behind It — Olive Song

The discussion details the engineering stack and open-source philosophy behind MiniMax's model, M3. Olive Song emphasizes that the open weights approach allows the community to build upon and optimize the model, fostering widespread intelligence access. The technical focus covers advanced training techniques—including multimodality (text, image, video) and Reinforcement Learning (RL) for long-horizon tasks like replicating academic papers (12-hour runs)—and the complex infrastructure required for deployment. Key engineering challenges discussed include writing specialized GPU kernels, optimizing the inference stack from 'day zero,' managing KV cache growth in agentic workflows, and adapting to shifting workloads from chat-based to multi-turn, tool-calling agents.

Key takeaways

Open Weights Philosophy 2:07
MiniMax advocates for open source because it aligns with their mission of making intelligence widely accessible. By releasing weights, they enable developers (like Together AI) to optimize the model's inference speed and capabilities through community contributions.
Multimodality Training 8:02
MiniMax M3 is multimodal, understanding text, code, images, and videos. Crucially, it was trained multimodally from scratch to prevent 'training collapse,' ensuring that the modalities naturally interact (e.g., visual tokens attending to text tokens).
Agentic Workloads and Inference Shifts 13:40
The workload is shifting from simple chat turns to complex agentic workflows involving hundreds of multi-turn tool calls. This requires significant optimization in the inference stack, particularly concerning KV cache management and routing.
Long-Horizon RL Tasks
Training for complex tasks (e.g., replicating an ICLR paper over 12 hours) requires careful formulation of the problem, defining environments, and optimizing reward functions within the Reinforcement Learning framework.

Watch on YouTube Full article