# What's New in Inference Engineering — Philip Kiely, Baseten

## Executive summary

This talk reviews the state-of-the-art in inference engineering, focusing on advancements in data center optimization. The speaker identifies three major areas—Quantization, Caching (KV compaction), and Speculation—that are rapidly evolving. A key theme is the blurring line between dedicated training and inference, where optimizations are increasingly derived from dedicated training processes. Significant advancements include the development of diffusion-based speculative decoding (DFlash) and techniques like KV compaction, which aim to maintain near-lossless memory retention for context.

## Key takeaways

- Two Types of Inference Engineering: Two distinct approaches emerged: Local inference (focusing on getting the model working on limited hardware via quantization, distillation, and pruning) and Data Center inference (focusing on getting the model working, then making it less slow, utilizing techniques like KV-aware routing and speculation).
- Training for Inference: Optimizations are increasingly coming from dedicated training processes, blurring the traditional line between training and inference. This creates a cycle: faster inference generates more data, which trains a better model, leading to even faster inference.
- Speculation Advancements: The field has moved from simple speculative decoding to advanced methods like DFlash (diffusion for speculation), which predicts multiple tokens (8 or 16) in a single forward pass, achieving a more than 3x improvement in acceptance rate compared to previous methods.

## Technical details

- Quantization: Quantization uses smaller, less precise number formats to save bandwidth and compute. While TurboQuant (a quantization approach using polar coordinates) was highly publicized for allowing KV cache compression down to four bits (saving half the memory/bandwidth), the speaker noted that the required additional decode computation cut Tokens Per Second (TPS) by more than half, making it unsuitable for most data center workloads. Current focus remains on NVFP4 quantization, emphasizing KV-aware routing and offloading.
- Caching (KV Compaction): The KV cache is critical for efficiency. While techniques like RAG and agent harnesses compress context, KV compaction aims for near-lossless compression of the cache itself. The 'Still' mechanism, developed by the Base10 research team, is highlighted: it uses a fixed set of learnable query vectors that cross-attend against the full KV cache to produce a compact set of keys and values in a single forward pass.
- Speculative Decoding: Speculative decoding uses draft tokens to generate multiple tokens per forward pass, improving TPS. The evolution includes: 1) Training a large model on the target's hidden states (replacing small family models); 2) DFlash (diffusion for speculation), which predicts sequences of tokens; and 3) DSpark (a newer, unreleased research method pairing diffusion with a sequential model). Continuous retraining on live prompts/responses can boost token acceptance rates by 20% to 2x.

## Practical implications

- Data center optimization is shifting toward integrating training and inference processes, requiring build engineers to consider model optimization pipelines that span both domains.
- The focus on KV cache management (compaction, offloading, and routing) is critical for scaling large context models.
- Adopting advanced speculative decoding techniques (like DFlash) can significantly boost throughput (TPS) and acceptance rates, requiring careful integration into the inference serving stack.

## Topics

Inference Engineering, Quantization, KV Cache, Speculative Decoding, LLM Optimization, Data Center AI, Inference Engineering (Book), TurboQuant, Still (KV Compaction)

Source: https://www.youtube.com/watch?v=75ckHC2LU_0