Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave
Summary
This talk details the architecture of a vertical mobility inference platform designed to serve diverse AI workloads, from small models to trillion-parameter scale. The platform addresses the challenge of efficiently serving four distinct workload shapes (agentic, chat, streaming, batch) while supporting two primary consumption models: serverless (pay-per-token) and dedicated (billed per GPU hour). Core architectural focus is placed on optimizing resource utilization through KV cache awareness, prefill/decode disaggregation, and advanced techniques like quantization and speculative decoding.
Key takeaways
-
Dual Consumption Models
2:02
The platform supports Serverless (pay-per-token, offering 'provisioned throughput' to mitigate noisy neighbor issues) and Dedicated (billed per GPU hour, offering full control over hardware and deployment).
-
Workload Shape Optimization
5:10
Four workload types—agentic, chat, streaming (voice/video), and batch—require different resource strategies. Agentic turns, in particular, demand low latency and efficient KV cache management.
-
KV Cache Management
11:57
The router is designed to be KV cache aware, prioritizing cache locality to avoid recomputing the expensive prefill step. For chat workloads, the KV cache is offloaded to high bandwidth storage rather than being evicted.
-
Performance Levers
Key performance optimizations include 4-bit quantization (e.g., NVFP4), speculative decoding (trained asynchronously on customer data), and the ability to disaggregate prefill and decode stages.
Technical details
-
Inference Architecture
515s
The platform routes requests through a gateway and control plane, supporting multiple inference engines (VLM, SLMs, etc.) across heterogeneous Nvidia GPUs in different zones.
-
Prefill Efficiency
717s
Since 80-90% of input tokens in agentic requests are often identical to previous requests, recomputing the prefill is highly expensive. Caching is critical, leading to differentiated pricing for input vs. cached tokens.
-
Dedicated Deployment
800s
Dedicated customers use a private gateway, ensuring isolation and billing based on GPU usage per hour, allowing them to manage their own model deployments and decide on prefill/decode disaggregation.
-
Batch Processing
900s
Batch workloads (which can run for hours) utilize capacity that is scheduled to run overnight, allowing real-time capacity during the day to be drained by batch processing at night.
Mentioned resources
- CoreWeave
- Nvidia GPUs
- NVFP4
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.