Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta
Summary
Inference workloads are rapidly becoming foundational, hyperscale infrastructure, outpacing even major microservices. The complexity has shifted from optimizing models or kernels to mastering the orchestration layer—the 'control plane.' This requires treating a request as a distributed transaction, necessitating sophisticated scheduling and reliability mechanisms that account for seven axes (e.g., GPU generation, KV cache state, tenant priority). The core optimization metric must shift from 'cost per token' to 'cost per successful task.'
Key takeaways
-
Inference as a Distributed Transaction
11:45
Unlike traditional RPC calls, inference involves multiple network hops (gateway, router, scheduler, runtime) where any step can retry, time out, or fail. Reliability must therefore be managed by the control plane, which sees the entire workflow, especially when partial failures occur (e.g., after streaming 200 tokens).
-
The Shift to Orchestration
3:50
The value has moved from simple models to the orchestration layer. The system must manage complex interactions, such as a routing decision changing the cache hit rate, which subsequently affects batch composition and GPU utilization. This coupling is the new complexity.
-
Optimization Metric: Cost per Successful Task
14:05
The goal of optimization is not merely minimizing cost per token or per request. The critical metric is 'cost per successful task,' as this reflects the actual value delivered to the user and accounts for retries, failures, and operational overhead.
-
The Need for a Dedicated Control Plane
17:00
As inference scales, it requires its own dedicated control plane, analogous to how Kubernetes managed VMs. This plane must manage resources beyond CPU/Memory, including GPU, KV cache, and token limits, to make optimal scheduling and routing decisions.
Technical details
-
Inference vs. Microservices Serving
350s
Microservices typically assume short, uniform requests and are often stateless. LLM inference, however, involves variable request shapes (50 to 100,000 tokens), requires continuous in-flight batching, and maintains a large, expensive per-request state (the KV cache).
-
Advanced Scheduling Requirements
600s
An inference scheduler must be aware of at least seven axes: GPU generation (e.g., H00, E00, B200), memory headroom, KV cache state, model weight status (warm/cold), tenant priority, latency budget, and the overall workflow context.
-
Optimization Framework (Four Quadrants)
730s
Optimizations can be categorized into four areas: 1) Avoid the work (caching); 2) Share the work (batching, e.g., continuous batching); 3) Move the work (routing to cheaper models/regions); or 4) Delay the work (admission control and queueing).
-
Observability as a Control Loop Input
900s
The platform must use observability metrics (Time to First Token, Utilization Ratio, End-to-End Race Latency) to feed a control loop that drives decisions in scheduling and routing, optimizing the trade-off between latency, cost, and throughput.
Mentioned resources
- Kubernetes
- VLM, Triton, SD Lang
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.