# Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta

## Executive summary

Inference workloads are rapidly becoming foundational, hyperscale infrastructure, outpacing even major microservices. The complexity has shifted from optimizing models or kernels to mastering the orchestration layer—the 'control plane.' This requires treating a request as a distributed transaction, necessitating sophisticated scheduling and reliability mechanisms that account for seven axes (e.g., GPU generation, KV cache state, tenant priority). The core optimization metric must shift from 'cost per token' to 'cost per successful task.'

## Key takeaways

- Inference as a Distributed Transaction: Unlike traditional RPC calls, inference involves multiple network hops (gateway, router, scheduler, runtime) where any step can retry, time out, or fail. Reliability must therefore be managed by the control plane, which sees the entire workflow, especially when partial failures occur (e.g., after streaming 200 tokens).
- The Shift to Orchestration: The value has moved from simple models to the orchestration layer. The system must manage complex interactions, such as a routing decision changing the cache hit rate, which subsequently affects batch composition and GPU utilization. This coupling is the new complexity.
- Optimization Metric: Cost per Successful Task: The goal of optimization is not merely minimizing cost per token or per request. The critical metric is 'cost per successful task,' as this reflects the actual value delivered to the user and accounts for retries, failures, and operational overhead.
- The Need for a Dedicated Control Plane: As inference scales, it requires its own dedicated control plane, analogous to how Kubernetes managed VMs. This plane must manage resources beyond CPU/Memory, including GPU, KV cache, and token limits, to make optimal scheduling and routing decisions.

## Technical details

- Inference vs. Microservices Serving: Microservices typically assume short, uniform requests and are often stateless. LLM inference, however, involves variable request shapes (50 to 100,000 tokens), requires continuous in-flight batching, and maintains a large, expensive per-request state (the KV cache).
- Advanced Scheduling Requirements: An inference scheduler must be aware of at least seven axes: GPU generation (e.g., H00, E00, B200), memory headroom, KV cache state, model weight status (warm/cold), tenant priority, latency budget, and the overall workflow context.
- Optimization Framework (Four Quadrants): Optimizations can be categorized into four areas: 1) Avoid the work (caching); 2) Share the work (batching, e.g., continuous batching); 3) Move the work (routing to cheaper models/regions); or 4) Delay the work (admission control and queueing).
- Observability as a Control Loop Input: The platform must use observability metrics (Time to First Token, Utilization Ratio, End-to-End Race Latency) to feed a control loop that drives decisions in scheduling and routing, optimizing the trade-off between latency, cost, and throughput.

## Practical implications

- Platform teams must focus on the end-to-end infrastructure and control plane, as the economics and reliability are determined by the system, not just the model.
- Implement workflow-aware orchestration to manage the cost and failure implications of multi-step agentic workflows.
- Design circuit breakers and admission controls at the routing layer to prevent cascading failures and manage retry budgets effectively.

## Topics

Distributed Systems, AI Infrastructure, LLM Serving, Control Plane, Scheduling, Cloud Computing, Kubernetes, VLM, Triton, SD Lang

Source: https://www.youtube.com/watch?v=7c9FSUVcXR0