The Video Signal

Topic

Speculative Decoding

All digests tagged Speculative Decoding

What's New in Inference Engineering — Philip Kiely, Baseten thumbnail

· 19:09

What's New in Inference Engineering — Philip Kiely, Baseten

This talk reviews the state-of-the-art in inference engineering, focusing on advancements in data center optimization. The speaker identifies three major areas—Quantization, Caching (KV compaction), and Speculation—that are rapidly evolving. A key theme is the blurring line between dedicated training and inference, where optimizations are increasingly derived from dedicated training processes. Significant advancements include the development of diffusion-based speculative decoding (DFlash) and techniques like KV compaction, which aim to maintain near-lossless memory retention for context.

Key takeaways

  1. Two Types of Inference Engineering 3:33

    Two distinct approaches emerged: Local inference (focusing on getting the model working on limited hardware via quantization, distillation, and pruning) and Data Center inference (focusing on getting the model working, then making it less slow, utilizing techniques like KV-aware routing and speculation).

  2. Training for Inference 5:52

    Optimizations are increasingly coming from dedicated training processes, blurring the traditional line between training and inference. This creates a cycle: faster inference generates more data, which trains a better model, leading to even faster inference.

  3. Speculation Advancements 10:50

    The field has moved from simple speculative decoding to advanced methods like DFlash (diffusion for speculation), which predicts multiple tokens (8 or 16) in a single forward pass, achieving a more than 3x improvement in acceptance rate compared to previous methods.

Watch on YouTube Full article

The Inference Frontier: 10x Faster Models to Self-Optimizing AI — Philip Kiely & Ali Taha, Baseten thumbnail

· 1:42:54

The Inference Frontier: 10x Faster Models to Self-Optimizing AI — Philip Kiely & Ali Taha, Baseten

The video provides a deep dive into modern inference engineering, detailing how large language models (LLMs) are optimized for production use to achieve massive speedups (up to 10x). Key techniques discussed include cache-aware routing, disaggregated prefill/decode, speculative decoding, and advanced quantization methods like NVFP4. The conversation also expands beyond LLMs into video generation architectures and the convergence of training and inference, emphasizing that modern AI infrastructure is becoming a complex blend of software optimization (kernels, runtimes) and hardware architecture (B200s, Blackwell).

Key takeaways

  1. Inference Optimization Stacking for 10x Gains 1:39:42

    Achieving massive speedups requires stacking multiple optimizations. A typical baseline might be 30-40 tokens/second; reaching 10x performance involves combining techniques like speculative decoding, quantization (e.g., BF-16 to NVFP4), cache-aware routing, and disaggregation of prefill/decode stages. The overall gain is multiplicative.

  2. The Role of Specialized Tooling (Dynamo) 1:22:40

    NVIDIA Dynamo is presented not as an out-of-the-box optimizer, but as a developer toolkit/library for coordinating complex inference tasks like KV cache offloading and prefill/decode disaggregation across heterogeneous hardware.

  3. Training vs. Inference Convergence 1:40:52

    The field is moving toward a continuous improvement loop where inference data (traces) are used to continually post-train, fine-tune, and optimize models. This unification means inference engineers must be highly familiar with training techniques like Quantization Aware Training (QAT).

  4. Video Generation Complexity 1:30:02

    Unlike text generation (which is autoregressive), video diffusion models are non-sequential. Generating long, coherent videos requires managing massive attention spans (e.g., attending to tens of thousands of tokens for just 5 seconds), making the quadratic complexity a major bottleneck.

Watch on YouTube Full article