Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher
Summary
This workshop provides a deep dive into optimizing Large Language Model (LLM) inference at scale. The core challenge is managing the high computational and memory costs associated with LLMs, particularly as context length and concurrent users increase. Solutions are categorized into two areas: model optimizations (e.g., quantization, Grouped Query Attention, Flash Attention) to reduce size/compute, and serving optimizations (e.g., Paged Attention, Continuous Batching, Prefix Caching) to improve throughput and memory efficiency. The session concludes with a comparative analysis of leading inference engines like vLLM and SGLang.
Key takeaways
-
Inference Cost Drivers
0:05
The primary pain points are: 1) Memory consumption that grows with context length (due to the KV cache); 2) Time To First Token (TTFT) degradation as input size increases; and 3) Throughput collapse when requests are processed sequentially.
-
Memory Scaling is Critical
0:10
The Key-Value (KV) cache memory usage scales significantly: for a Mistral 7B model, the KV size per token is approximately 131 KB. Serving multiple users with long contexts quickly exhausts GPU VRAM (e.g., 80 concurrent users and 4K context can require 42 GB).
-
Serving Optimizations are Essential
0:24
To maximize throughput, techniques like Paged Attention (borrowed from OS paging) solve memory fragmentation; Continuous Batching keeps the GPU occupied by processing multiple requests efficiently; and Prefix Caching saves computation when handling similar prompts across different users.
-
Engine Selection Trade-offs
0:46
While vLLM and SGLang show no statistical difference on standard workloads, SGLang demonstrated a significant advantage (three to four times better) when handling complex agentic branching workflows.
Technical details
-
Attention Mechanism & KV Cache
13s
The attention layer is the most compute-intensive part of the transformer. It requires projecting every token into Key (K), Query (Q), and Value (V) vectors. The resulting K/V matrices are stored in the cache, which grows linearly with context length and concurrently with the number of users.
-
Model Optimization Techniques
32s
Techniques to reduce model size or computation include: Quantization (e.g., FP16 $ ightarrow$ Int8/Int4) for memory reduction; Multi-Head Attention (MHA), Grouped Query Attention (GQA), and Multi-Head Latent Attention (MLLA) which reduce the number of K/V heads to save compute; Flash Attention, which tiles matrices into small blocks processed by Tensor Cores to improve speed.
-
Inference Phases and Bottlenecks
25s
LLM inference has two phases: Prefill (compute-bound, determines TTFT) and Decode (memory-bound, determines inter-token latency). The decode phase is limited by the bandwidth of transferring K/V vectors from High Bandwidth Memory (HBM) to Shared Memory.
-
Advanced Serving Concepts
40s
Speculative Decoding uses a smaller 'teacher' model to predict several tokens, which are then verified by the main model, accelerating generation. Radix Tree is an advanced prefix tree used for efficient caching of similar prompts in agentic workflows.
Mentioned resources
- LLM inference at scale (Repo)
- Slides and Jupyter Notebooks
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.