The Video Signal

Topic

Quantization

All digests tagged Quantization

What's New in Inference Engineering — Philip Kiely, Baseten thumbnail

· 19:09

What's New in Inference Engineering — Philip Kiely, Baseten

This talk reviews the state-of-the-art in inference engineering, focusing on advancements in data center optimization. The speaker identifies three major areas—Quantization, Caching (KV compaction), and Speculation—that are rapidly evolving. A key theme is the blurring line between dedicated training and inference, where optimizations are increasingly derived from dedicated training processes. Significant advancements include the development of diffusion-based speculative decoding (DFlash) and techniques like KV compaction, which aim to maintain near-lossless memory retention for context.

Key takeaways

  1. Two Types of Inference Engineering 3:33

    Two distinct approaches emerged: Local inference (focusing on getting the model working on limited hardware via quantization, distillation, and pruning) and Data Center inference (focusing on getting the model working, then making it less slow, utilizing techniques like KV-aware routing and speculation).

  2. Training for Inference 5:52

    Optimizations are increasingly coming from dedicated training processes, blurring the traditional line between training and inference. This creates a cycle: faster inference generates more data, which trains a better model, leading to even faster inference.

  3. Speculation Advancements 10:50

    The field has moved from simple speculative decoding to advanced methods like DFlash (diffusion for speculation), which predicts multiple tokens (8 or 16) in a single forward pass, achieving a more than 3x improvement in acceptance rate compared to previous methods.

Watch on YouTube Full article

GPU Kernel Formal Verification thumbnail

· 34:39

GPU Kernel Formal Verification

This talk details the critical challenge of establishing trust in high-performance GPU kernels generated by AI. While AI tools like KForge can efficiently generate optimized kernels (e.g., replacing expensive divide operations with exponent/mantissa decomposition), the primary concern shifts from performance generation to correctness. The speaker introduces Tatwa, a tensor algebra equivalence checker, which uses formal verification (Z3 solvers) to reason about the semantic equivalence between a reference kernel (e.g., PyTorch model) and an AI-generated candidate kernel. The talk highlights that while testing is useful, only formal verification can detect subtle bugs, such as missing intermediate clamp operations or semantic discrepancies, by reasoning about the underlying mathematical primitives.

Key takeaways

  1. Limitations of Testing vs. Verification 18:13

    Testing is inherently sampling and can only prove the absence of bugs for tested inputs. Formal verification, however, reasons about the semantics and mathematical structure, allowing it to detect bugs (like missing clamps or incorrect scaling factors) that only manifest for specific, untested inputs.

  2. The Need for Semantic Equivalence Checking 22:10

    The core problem is ensuring that the optimized AI-generated kernel maintains the exact mathematical semantics of the original reference kernel. Tatwa achieves this by decomposing high-level operations (like `clamp` or `softmax`) into basic mathematical primitives for the Z3 solver to reason over.

  3. Bounded Verification Approach 27:10

    Due to computational complexity, the current approach uses bounded verification, limiting the summation or expansion of product terms (e.g., capping the reduction over k to a small value) to keep the problem solvable by solvers like Z3.

Watch on YouTube Full article

BF16 vs NVFP4 with Nemotron 3.5 Lightning thumbnail

· 9:20

BF16 vs NVFP4 with Nemotron 3.5 Lightning

This technical deep dive compares two model quantization formats, BF16 and NVFP4, using Nemotron 3.5 Lightning. NVFP4 is presented as a highly efficient, low-precision format that significantly reduces memory footprint and increases throughput for inference. While NVFP4 is recommended for token generation, BF16 or full precision checkpoints are advised for model customization, fine-tuning, or training stability. The technology has expanded, allowing NVFP4 to run efficiently on architectures like Hopper and Ampere, not just Blackwell.

Key takeaways

  1. NVFP4 for Inference Efficiency

    Using NVFP4 drastically reduces the memory footprint required to store model weights compared to BF16, making deployment on smaller hardware more feasible. It is recommended for high-speed token generation.

  2. BF16 for Training Stability 3:56

    While NVFP4 is ideal for inference, BF16 or full precision checkpoints should be used if the developer plans to customize the model, fine-tune weights, or use custom quantization algorithms, as this ensures better training stability.

  3. Minimal Accuracy Loss 2:30

    Despite the quantization from BF16 to NVFP4, the accuracy degradation is reported to be very small (e.g., 99.99% retained), thanks to intrinsic properties of model weights.

  4. Broad Hardware Compatibility 5:40

    The ability to run NVFP4 checkpoints is no longer limited to the newest hardware (Blackwell); it is now supported across a wide range of GPUs, including Hopper and Ampere.

Watch on YouTube Full article

Run Local Models in Pi: llama.cpp, GGUF, and the /llama Command thumbnail

· 6:11

Run Local Models in Pi: llama.cpp, GGUF, and the /llama Command

This tutorial provides a complete guide on running large language models (LLMs) locally on a Raspberry Pi using `llama.cpp`. The process emphasizes privacy and offline capability by ensuring that no prompts, code, or data leave the local machine. Key steps include installing `llama.cpp`, selecting an optimal GGUF model (like Qwen3 8B) based on hardware compatibility, and loading/running the model via the `/llama` command.

Key takeaways

  1. Local Model Operation

    Running models locally with `llama.cpp` ensures that all data processing remains entirely within the machine, eliminating reliance on third-party APIs for prompts, code, or data (0:15).

  2. Installation and Setup 2:29

    Install `llama.cpp` using the provided installer link (`llama.app`) to establish the local server environment, allowing subsequent model interaction via the `llama serve` command (0:59).

  3. Model Selection and Quantization 3:35

    To select an optimal model, use the hardware compatibility feature on sites like `llama.app`. This tool recommends the best quantization level (e.g., 4-bit) for specific hardware (M4 Max), which is crucial for performance (2:30).

  4. Running Models via Pi 6:00

    After downloading a model ID and selecting the appropriate quantization (e.g., Q4), models can be loaded and interacted with directly using the `/llama` command within the local environment (3:30).

Watch on YouTube Full article

Compression at the Edge — Chris Alexiuk, NVIDIA thumbnail

· 46:01

Compression at the Edge — Chris Alexiuk, NVIDIA

This panel discusses model compression techniques—primarily quantization—that enable running massive Large Language Models (LLMs) on resource-constrained edge devices. Key advancements include formats like NVFP4 and the ability to shrink models dramatically (e.g., GLM 5.2 from 1.5 TB to 250 GB). The discussion emphasizes that successful compression requires understanding model architecture, using advanced methods like Quantization Aware Distillation (QAD), and prioritizing evaluation metrics such as KL divergence over simple accuracy scores.

Key takeaways

  1. Model Compression is Critical for Edge AI

    Compression techniques are essential to democratize LLMs, making them viable for local deployment on consumer hardware (e.g., laptops/phones). The goal is enabling powerful models to run without constant reliance on cloud APIs.

  2. Advanced Quantization Formats and Techniques 0:04

    NVIDIA's NVFP4 is a specialized 4-bit float format where every group of 16 values shares one FP8 scale. For large models (>20B parameters), Post-Training Quantization (PTQ) works well, while smaller models (<20B) require Quantization Aware Distillation (QAD).

  3. Evaluation Focus Shifts to Logits and Architecture 0:08

    Verifying model integrity after compression is complex. The Super Weights paper suggests that quantizing even one number can degrade performance by 20%. Therefore, the preferred signal for evaluation is KL divergence between BF16 and quantized output logits, rather than traditional accuracy scores.

Watch on YouTube Full article

Llama.cpp vs vLLM: Which Local LLM Engine Actually Scales? thumbnail

· 10:36

Llama.cpp vs vLLM: Which Local LLM Engine Actually Scales?

The video compares Llama.cpp and vLLM, two leading engines for running Large Language Models (LLMs) locally. Llama.cpp is optimized for accessibility on consumer hardware (CPU/GPU), utilizing techniques like quantization and the GGUF format to run models efficiently on personal devices or edge environments. Conversely, vLLM focuses on maximizing efficiency at production scale, supporting diverse accelerators (NVIDIA, TPU, etc.) and implementing advanced optimizations such as continuous batching and paged attention for high-throughput workloads in cloud or Kubernetes deployments.

Key takeaways

  1. Llama.cpp Use Case

    Ideal for running LLMs on consumer hardware (laptops, Raspberry Pi) or edge devices due to its focus on accessibility and CPU/GPU optimization. Key features include quantization (reducing precision from FP16 to INT8/INT4) and packaging models into a single .gguf file.

  2. vLLM Use Case 4:10

    Designed for high-throughput, production-scale workloads in environments like VMs or Kubernetes. It supports diverse hardware accelerators (NVIDIA GPUs, TPUs, etc.) and advanced features like continuous batching and paged attention to manage KV cache efficiently.

  3. Model Deployment Strategy 8:10

    The choice depends on the environment: use Llama.cpp for personal/offline use cases, and vLLM when deploying in a high-performance, multi-user production setting.

Watch on YouTube Full article

Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google thumbnail

· 21:45

Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google

The deployment of AI on edge devices (robotics, mobile phones) is constrained primarily by DRAM cost, necessitating the development of tiny LLMs. The talk details strategies—including aggressive quantization and fine-tuning—to shrink models like Gemma to run efficiently on low-power hardware (e.g., Raspberry Pi, Qualcomm NPUs). Tiny models (<500M parameters) are ideal for wide-scale deployment but often require specialized tasks (ASR, vision, function calling) and robust synthetic data generation for fine-tuning.

Key takeaways

  1. DRAM Cost is the Primary Edge Constraint 1:45

    The main challenge in edge AI deployment is not compute power, but DRAM cost. Mobile phone manufacturers are reducing on-device RAM, making model size critical. (3:25)

  2. Quantization Enables Small Footprints 6:08

    Models can be aggressively quantized (e.g., Gemma to 2.9 bits per weight) and optimized using techniques like per-layer embeddings to minimize memory footprint, allowing deployment on devices with limited RAM. (6:08)

  3. Tiny Models Require Fine-Tuning 14:20

    While small models are easy to use via zero-shot prompting, achieving high reliability for specific tasks like function calling or voice dictation requires fine-tuning using synthetic data sets. (14:28)

  4. Voice-to-Function Calling is Key 16:00

    Fine-tuning a small model to handle both ASR and function calling allows for robust, offline interaction on low-tier IoT/edge devices, which is crucial where complex UIs are difficult. (14:28)

Watch on YouTube Full article

Local AI 201 thumbnail

· 53:22

Local AI 201

The session provides an advanced deep dive into local AI deployment, emphasizing that successful LLM inference is not determined by hardware capacity alone. Instead, it requires selecting a balanced stack comprising the right model, quantization level, and specialized inference engine (e.g., VLLM, llama.cpp) for the specific use case—whether single-user chat or high-concurrency agentic workflows. Key performance metrics like memory bandwidth are shown to be more critical than raw memory capacity when scaling up requests.

Key takeaways

  1. Start with the Use Case, Not the Hardware 2:09

    When designing a local AI solution, always begin by defining the required use case (e.g., single-user chatbot vs. 50-person agentic workflow). The hardware, model, and engine stack must then be selected to support that specific requirement.

  2. Memory Bandwidth is Critical for Throughput 4:08

    For serving multiple requests (high throughput), memory bandwidth is often a more critical bottleneck than total memory capacity. For example, the RTX 5090 was shown to achieve significantly higher performance due to its high bandwidth compared to other devices.

  3. Engine Selection Dictates Performance Under Load 5:41

    The choice of inference engine (e.g., VLLM vs. llama.cpp) and kernel optimization is paramount. Improperly selecting an engine can severely limit performance, causing a high-bandwidth device to perform worse than a lower-bandwidth machine under load.

  4. Local AI Offers Superior Privacy and Control 7:30

    Running LLMs locally provides massive advantages in security, privacy, and control compared to relying on third-party cloud APIs. This allows users to fully tune the stack for long-term stability.

Watch on YouTube Full article

New Model: Inkling by Thinking Machine on Hugging Face thumbnail

· 36:00

New Model: Inkling by Thinking Machine on Hugging Face

Thinking Machines announced Inkling, a massive open model with nearly one trillion parameters. It is designed as a natively multimodal architecture, accepting image, text, and audio inputs simultaneously. Key technical features include a Mixture-of-Experts (MoE) structure (975B total / 41B active params), a 1M token context window, and advanced deployment support across multiple frameworks like `transformers`, SGLang, vLLM, and `llama.cpp`. The model is available in BF16 and NVFP4 formats, facilitating high-performance inference on various hardware setups.

Key takeaways

  1. Multimodal Capability & Scale 3:50

    Inkling is a true multimodal model that natively processes image, text, and audio inputs using a single architecture. It boasts an immense 1M token context window and was trained on 45T tokens.

  2. Architectural Innovations 5:10

    The model utilizes relative attention (replacing RoPE) and a specialized SConv layer to efficiently aggregate hidden states, making it highly efficient for multimodal tasks compared to previous models.

  3. Deployment Flexibility 7:10

    Inkling supports multiple deployment paths: BF16 (requiring ~2TB VRAM) and NVFP4 (600GB VRAM). It provides day-zero support for `transformers`, SGLang, vLLM, and `llama.cpp` (including GGUF quants via Unsloth), enabling diverse inference environments.

Watch on YouTube Full article