What's New in Inference Engineering — Philip Kiely, Baseten
Summary
This talk reviews the state-of-the-art in inference engineering, focusing on advancements in data center optimization. The speaker identifies three major areas—Quantization, Caching (KV compaction), and Speculation—that are rapidly evolving. A key theme is the blurring line between dedicated training and inference, where optimizations are increasingly derived from dedicated training processes. Significant advancements include the development of diffusion-based speculative decoding (DFlash) and techniques like KV compaction, which aim to maintain near-lossless memory retention for context.
Key takeaways
-
Two Types of Inference Engineering
3:33
Two distinct approaches emerged: Local inference (focusing on getting the model working on limited hardware via quantization, distillation, and pruning) and Data Center inference (focusing on getting the model working, then making it less slow, utilizing techniques like KV-aware routing and speculation).
-
Training for Inference
5:52
Optimizations are increasingly coming from dedicated training processes, blurring the traditional line between training and inference. This creates a cycle: faster inference generates more data, which trains a better model, leading to even faster inference.
-
Speculation Advancements
10:50
The field has moved from simple speculative decoding to advanced methods like DFlash (diffusion for speculation), which predicts multiple tokens (8 or 16) in a single forward pass, achieving a more than 3x improvement in acceptance rate compared to previous methods.
Technical details
-
Quantization
390s
Quantization uses smaller, less precise number formats to save bandwidth and compute. While TurboQuant (a quantization approach using polar coordinates) was highly publicized for allowing KV cache compression down to four bits (saving half the memory/bandwidth), the speaker noted that the required additional decode computation cut Tokens Per Second (TPS) by more than half, making it unsuitable for most data center workloads. Current focus remains on NVFP4 quantization, emphasizing KV-aware routing and offloading.
-
Caching (KV Compaction)
490s
The KV cache is critical for efficiency. While techniques like RAG and agent harnesses compress context, KV compaction aims for near-lossless compression of the cache itself. The 'Still' mechanism, developed by the Base10 research team, is highlighted: it uses a fixed set of learnable query vectors that cross-attend against the full KV cache to produce a compact set of keys and values in a single forward pass.
-
Speculative Decoding
570s
Speculative decoding uses draft tokens to generate multiple tokens per forward pass, improving TPS. The evolution includes: 1) Training a large model on the target's hidden states (replacing small family models); 2) DFlash (diffusion for speculation), which predicts sequences of tokens; and 3) DSpark (a newer, unreleased research method pairing diffusion with a sequential model). Continuous retraining on live prompts/responses can boost token acceptance rates by 20% to 2x.
Mentioned resources
- Inference Engineering (Book)
- TurboQuant
- Still (KV Compaction)
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.