# Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave

## Executive summary

This talk details the architecture of a vertical mobility inference platform designed to serve diverse AI workloads, from small models to trillion-parameter scale. The platform addresses the challenge of efficiently serving four distinct workload shapes (agentic, chat, streaming, batch) while supporting two primary consumption models: serverless (pay-per-token) and dedicated (billed per GPU hour). Core architectural focus is placed on optimizing resource utilization through KV cache awareness, prefill/decode disaggregation, and advanced techniques like quantization and speculative decoding.

## Key takeaways

- Dual Consumption Models: The platform supports Serverless (pay-per-token, offering 'provisioned throughput' to mitigate noisy neighbor issues) and Dedicated (billed per GPU hour, offering full control over hardware and deployment).
- Workload Shape Optimization: Four workload types—agentic, chat, streaming (voice/video), and batch—require different resource strategies. Agentic turns, in particular, demand low latency and efficient KV cache management.
- KV Cache Management: The router is designed to be KV cache aware, prioritizing cache locality to avoid recomputing the expensive prefill step. For chat workloads, the KV cache is offloaded to high bandwidth storage rather than being evicted.
- Performance Levers: Key performance optimizations include 4-bit quantization (e.g., NVFP4), speculative decoding (trained asynchronously on customer data), and the ability to disaggregate prefill and decode stages.

## Technical details

- Inference Architecture: The platform routes requests through a gateway and control plane, supporting multiple inference engines (VLM, SLMs, etc.) across heterogeneous Nvidia GPUs in different zones.
- Prefill Efficiency: Since 80-90% of input tokens in agentic requests are often identical to previous requests, recomputing the prefill is highly expensive. Caching is critical, leading to differentiated pricing for input vs. cached tokens.
- Dedicated Deployment: Dedicated customers use a private gateway, ensuring isolation and billing based on GPU usage per hour, allowing them to manage their own model deployments and decide on prefill/decode disaggregation.
- Batch Processing: Batch workloads (which can run for hours) utilize capacity that is scheduled to run overnight, allowing real-time capacity during the day to be drained by batch processing at night.

## Practical implications

- The choice between serverless and dedicated deployment models depends on whether the customer prioritizes ease-of-use (serverless) or granular control over hardware and billing (dedicated).
- For maximizing efficiency, customers should leverage cache-aware routing and consider advanced optimizations like speculative decoding and prefill/decode disaggregation.
- The platform design must accommodate the vastly different latency requirements of streaming (voice/video) versus the high throughput needs of batch processing.

## Topics

AI Inference, LLM Deployment, Cloud Computing, System Architecture, Optimization, CoreWeave, Nvidia GPUs, NVFP4

Source: https://www.youtube.com/watch?v=cQQbJqvZkpo