# KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat

## Executive summary

This talk details advanced strategies for optimizing LLM inference in complex agentic workloads, moving beyond the limitations of steady-state public benchmarks. The focus is on two critical levers: KV Cache-Aware Routing and Prefill/Decode (P/D) Disaggregation. Implementing these techniques—using frameworks like LLMD—significantly improves latency and throughput by managing volatile cache usage and separating compute-bound prefill from memory-bandwidth-hungry decode phases, particularly in the middle concurrency band.

## Key takeaways

- Agentic Workloads vs. Benchmarks: Real-world agentic workloads exhibit chaotic multi-turn interactions (up to 3,000 turns) with high cache hit rates (>90%) and massive input/output ratios (often >100:1), which standard public benchmarks fail to capture [0:00], [3:29].
- KV Cache Routing Optimization: Implementing KV cache-aware routing (via Endpoint Picker) is a cost-effective optimization, as the token cost difference between cached and uncached tokens can be as high as 10x [5:12]. This helps solve Time to First Token (TTFT) issues.
- P/D Disaggregation Benefits: Separating prefill and decode into independent, scalable pods prevents 'phase interference'—where a long prefill stalls token generation (decode)—leading to drastically reduced P99 Inter Token Latency (ITL) from ~900ms down to ~100ms [9:28], [10:46].
- Prerequisites for PD: Effective P/D disaggregation requires an advanced, high-speed network fabric like RDMA or RoCE to facilitate the transfer of KV caches between prefill and decode workers [10:46]. If such a fabric is unavailable, aggregated serving may be preferable.

## Technical details

- LLM Inference Frameworks: The discussion centers on LLMD (Distributed LM LLM inference framework), a Kubernetes-native, unified control plane designed for agentic workloads. Key components include the Endpoint Picker (EP) for optimal pod routing and specialized workload APIs like 'disaggregated set' [10:28].
- P/D Architecture Mechanics: In P/D, the pre-fill worker processes the prompt and constructs the initial KV cache, which is then transferred across the network fabric using specialized metadata to the dedicated decode worker. This separation allows independent scaling of both phases [10:46].
- Performance Metrics and Scaling: The architecture must dynamically balance multiple knobs, including SLO targets, QEPs (Quality Experience Points), KV cache locality metrics, and PD ratios. The system requires dynamic autoscaling to independently scale prefill and decode pools [12:00].

## Practical implications

- When designing for agentic workloads, prioritize optimizing for interactive latency and TTFT over raw steady-state throughput.
- Assess the network infrastructure (RDMA/RoCE) before implementing P/D disaggregation; this hardware is critical for cache transfer performance.
- Utilize specialized frameworks like LLMD to manage complex multi-dimensional scaling requirements, including independent autoscaling of prefill and decode pools.

## Topics

LLM Inference, Kubernetes, KV Cache Management, Distributed Systems, Agentic Workloads, Performance Engineering, Free Course on Deep Learning AI, Red Hat Developer Portal Blogs (Distributed Inference)

Source: https://www.youtube.com/watch?v=YXowceUKYJI