# Local AI 202: Get Massive Speed-ups with Speculative Decoding

## Executive summary

This technical deep dive introduces speculative decoding techniques (like MTP, DFlash, and EAGLE) designed to achieve massive speed-ups in Large Language Model (LLM) inference, particularly during the decode phase. The process involves training a smaller, highly efficient 'draft model' (speculator) to predict multiple candidate tokens in parallel. These candidates are then validated by the larger, target LLM in a single forward pass, significantly reducing memory bandwidth consumption and improving throughput. The presentation covers the entire lifecycle, from data preparation and quantization to advanced training techniques and deployment considerations.

## Key takeaways

- Speculative Decoding Mechanism: Speculative decoding works by having a small 'draft model' propose multiple candidate tokens (e.g., 4 to 16 tokens) which are then validated by the large target model in one forward pass. This process dramatically reduces the computational cost associated with sequential token generation, leading to significant throughput gains (e.g., up to 4x).
- Training and Optimization Pipeline: Training a speculator requires a multi-step pipeline: 1) Data preparation (generating data using the target model); 2) Hidden state extraction (offline or online); 3) Training the speculator (using advanced optimizers like Muon/AdamW); and 4) Quantization (e.g., quantizing hidden states down to 4 bits) to manage massive disk storage requirements.
- Architecture and Efficiency: Advanced techniques like Multi-Token Prediction (MTP) for Gemma 4 utilize the target model's existing architecture (e.g., local/global attention layers) to minimize the size of the draft model. This allows the speculator to leverage the target model's pre-computed activations (KV cache) for efficient prediction.
- Deployment Considerations: When deploying, the 'block size' (number of tokens speculated) is a critical hyperparameter. Adaptive verification methods are recommended to dynamically adjust the number of tokens verified based on the model's certainty, optimizing throughput in high-concurrency environments.

## Technical details

- Inference Process: LLM inference consists of two stages: Prefill (processing the initial prompt) and Decode (generating tokens sequentially). The decode phase is compute-bound due to the constant reading of weights and the KV cache, which speculative decoding aims to mitigate.
- Speculative Decoding Techniques: Various techniques exist, including classical speculative decoding, MTP (Multi-Token Prediction), DFlash, and EAGLE. These methods allow the draft model to predict multiple tokens simultaneously, which is superior to single-token prediction.
- Model Architecture Adaptation (MTP): For constrained environments (e.g., mobile), the draft model must mirror the target model's architecture (e.g., Gemma 4's interleaving of local and global attention layers) to efficiently utilize the target model's activations and KV cache.
- Optimization and Training: Training speculators benefits from using advanced optimizers like Muon over AdamW, which provides better convergence for speculative decoding. Furthermore, replacing full attention with Sliding Window Attention can improve stability and performance at longer sequence lengths.
- Quantization and Memory Management: To manage the massive disk storage required for hidden states, techniques like quantizing hidden states (e.g., down to 4 bits) are necessary. This significantly reduces storage requirements while maintaining high accuracy.

## Practical implications

- Build engineers can achieve substantial performance gains in LLM inference by implementing speculative decoding, moving beyond simple quantization.
- The training pipeline requires careful management of data generation (using large models to train smaller ones) and resource-intensive hidden state extraction.
- Deployment optimization involves tuning the 'block size' and implementing adaptive verification to maximize throughput in real-world, high-concurrency scenarios.
- Using specialized tools like `llama.app` can abstract away complex low-level optimizations, making local AI deployment easier.

## Topics

Large Language Models (LLMs), Speculative Decoding, Inference Optimization, Quantization, Transformer Architectures, Machine Learning Training Pipelines, llama.app, llama.cpp, speculators library, Hugging Face Hub, VLM project

Source: https://www.youtube.com/watch?v=H4azmWcqO5g