What Is an Inference Engine, Anyway? — Charles Frye, Modal
This talk provides a deep dive into the architecture and performance engineering of inference engines, which are critical components for running large language and multimodal models. The speaker details how these engines process requests through distinct stages—pre-processing (tokenization), core inference (scheduler, model runner), and post-processing (detokenization). Key performance bottlenecks are identified in the scheduler and memory bandwidth, leading to advanced techniques like KV caching, CUDA graphs, and speculative decoding to maximize throughput and minimize latency.
Key takeaways
-
Inference is a Revenue Center
3:35
Unlike training, which is a cost center, inference is a revenue-generating system, making it a critical focus for infrastructure engineering. Demand exists for setting up and optimizing proprietary inference stacks.
-
Workloads are Defined by Two Sub-Workloads
7:20
From an engine perspective, every workload involves two semi-independent sub-workloads: the 'prefill' phase (processing the input prompt) and the 'decode' phase (generating output tokens). These phases have fundamentally different arithmetic intensities.
-
The Scheduler is a Potential Bottleneck
5:40
While the GPU/accelerator is the most computationally intensive component, the scheduler process (which defines and schedules work to the GPU) can become a critical bottleneck because it manages resources and coordinates work, even if it performs minimal computation itself.
-
Performance Optimization is Multi-Layered
10:20
Optimization requires techniques at multiple levels: using KV caching for prefix reuse, employing CUDA graphs to reduce CPU overhead, and utilizing speculative decoding to parallelize the sequential token generation process.