Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta
Inference workloads are rapidly becoming foundational, hyperscale infrastructure, outpacing even major microservices. The complexity has shifted from optimizing models or kernels to mastering the orchestration layer—the 'control plane.' This requires treating a request as a distributed transaction, necessitating sophisticated scheduling and reliability mechanisms that account for seven axes (e.g., GPU generation, KV cache state, tenant priority). The core optimization metric must shift from 'cost per token' to 'cost per successful task.'
Key takeaways
-
Inference as a Distributed Transaction
11:45
Unlike traditional RPC calls, inference involves multiple network hops (gateway, router, scheduler, runtime) where any step can retry, time out, or fail. Reliability must therefore be managed by the control plane, which sees the entire workflow, especially when partial failures occur (e.g., after streaming 200 tokens).
-
The Shift to Orchestration
3:50
The value has moved from simple models to the orchestration layer. The system must manage complex interactions, such as a routing decision changing the cache hit rate, which subsequently affects batch composition and GPU utilization. This coupling is the new complexity.
-
Optimization Metric: Cost per Successful Task
14:05
The goal of optimization is not merely minimizing cost per token or per request. The critical metric is 'cost per successful task,' as this reflects the actual value delivered to the user and accounts for retries, failures, and operational overhead.
-
The Need for a Dedicated Control Plane
17:00
As inference scales, it requires its own dedicated control plane, analogous to how Kubernetes managed VMs. This plane must manage resources beyond CPU/Memory, including GPU, KV cache, and token limits, to make optimal scheduling and routing decisions.