Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave
This talk details the architecture of a vertical mobility inference platform designed to serve diverse AI workloads, from small models to trillion-parameter scale. The platform addresses the challenge of efficiently serving four distinct workload shapes (agentic, chat, streaming, batch) while supporting two primary consumption models: serverless (pay-per-token) and dedicated (billed per GPU hour). Core architectural focus is placed on optimizing resource utilization through KV cache awareness, prefill/decode disaggregation, and advanced techniques like quantization and speculative decoding.
Key takeaways
-
Dual Consumption Models
2:02
The platform supports Serverless (pay-per-token, offering 'provisioned throughput' to mitigate noisy neighbor issues) and Dedicated (billed per GPU hour, offering full control over hardware and deployment).
-
Workload Shape Optimization
5:10
Four workload types—agentic, chat, streaming (voice/video), and batch—require different resource strategies. Agentic turns, in particular, demand low latency and efficient KV cache management.
-
KV Cache Management
11:57
The router is designed to be KV cache aware, prioritizing cache locality to avoid recomputing the expensive prefill step. For chat workloads, the KV cache is offloaded to high bandwidth storage rather than being evicted.
-
Performance Levers
Key performance optimizations include 4-bit quantization (e.g., NVFP4), speculative decoding (trained asynchronously on customer data), and the ability to disaggregate prefill and decode stages.