KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat
Summary
This talk details advanced strategies for optimizing LLM inference in complex agentic workloads, moving beyond the limitations of steady-state public benchmarks. The focus is on two critical levers: KV Cache-Aware Routing and Prefill/Decode (P/D) Disaggregation. Implementing these techniques—using frameworks like LLMD—significantly improves latency and throughput by managing volatile cache usage and separating compute-bound prefill from memory-bandwidth-hungry decode phases, particularly in the middle concurrency band.
Key takeaways
-
Agentic Workloads vs. Benchmarks
5:29
Real-world agentic workloads exhibit chaotic multi-turn interactions (up to 3,000 turns) with high cache hit rates (>90%) and massive input/output ratios (often >100:1), which standard public benchmarks fail to capture [0:00], [3:29].
-
KV Cache Routing Optimization
10:28
Implementing KV cache-aware routing (via Endpoint Picker) is a cost-effective optimization, as the token cost difference between cached and uncached tokens can be as high as 10x [5:12]. This helps solve Time to First Token (TTFT) issues.
-
P/D Disaggregation Benefits
15:28
Separating prefill and decode into independent, scalable pods prevents 'phase interference'—where a long prefill stalls token generation (decode)—leading to drastically reduced P99 Inter Token Latency (ITL) from ~900ms down to ~100ms [9:28], [10:46].
-
Prerequisites for PD
17:26
Effective P/D disaggregation requires an advanced, high-speed network fabric like RDMA or RoCE to facilitate the transfer of KV caches between prefill and decode workers [10:46]. If such a fabric is unavailable, aggregated serving may be preferable.
Technical details
-
LLM Inference Frameworks
628s
The discussion centers on LLMD (Distributed LM LLM inference framework), a Kubernetes-native, unified control plane designed for agentic workloads. Key components include the Endpoint Picker (EP) for optimal pod routing and specialized workload APIs like 'disaggregated set' [10:28].
-
P/D Architecture Mechanics
1046s
In P/D, the pre-fill worker processes the prompt and constructs the initial KV cache, which is then transferred across the network fabric using specialized metadata to the dedicated decode worker. This separation allows independent scaling of both phases [10:46].
-
Performance Metrics and Scaling
1200s
The architecture must dynamically balance multiple knobs, including SLO targets, QEPs (Quality Experience Points), KV cache locality metrics, and PD ratios. The system requires dynamic autoscaling to independently scale prefill and decode pools [12:00].
Mentioned resources
- Free Course on Deep Learning AI
- Red Hat Developer Portal Blogs (Distributed Inference)
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.