What's New in Inference Engineering — Philip Kiely, Baseten
This talk reviews the state-of-the-art in inference engineering, focusing on advancements in data center optimization. The speaker identifies three major areas—Quantization, Caching (KV compaction), and Speculation—that are rapidly evolving. A key theme is the blurring line between dedicated training and inference, where optimizations are increasingly derived from dedicated training processes. Significant advancements include the development of diffusion-based speculative decoding (DFlash) and techniques like KV compaction, which aim to maintain near-lossless memory retention for context.
Key takeaways
-
Two Types of Inference Engineering
3:33
Two distinct approaches emerged: Local inference (focusing on getting the model working on limited hardware via quantization, distillation, and pruning) and Data Center inference (focusing on getting the model working, then making it less slow, utilizing techniques like KV-aware routing and speculation).
-
Training for Inference
5:52
Optimizations are increasingly coming from dedicated training processes, blurring the traditional line between training and inference. This creates a cycle: faster inference generates more data, which trains a better model, leading to even faster inference.
-
Speculation Advancements
10:50
The field has moved from simple speculative decoding to advanced methods like DFlash (diffusion for speculation), which predicts multiple tokens (8 or 16) in a single forward pass, achieving a more than 3x improvement in acceptance rate compared to previous methods.