Model Compression

Sep 14, 2026 · 9:20

BF16 vs NVFP4 with Nemotron 3.5 Lightning

This technical deep dive compares two model quantization formats, BF16 and NVFP4, using Nemotron 3.5 Lightning. NVFP4 is presented as a highly efficient, low-precision format that significantly reduces memory footprint and increases throughput for inference. While NVFP4 is recommended for token generation, BF16 or full precision checkpoints are advised for model customization, fine-tuning, or training stability. The technology has expanded, allowing NVFP4 to run efficiently on architectures like Hopper and Ampere, not just Blackwell.

Key takeaways

1. NVFP4 for Inference Efficiency

   Using NVFP4 drastically reduces the memory footprint required to store model weights compared to BF16, making deployment on smaller hardware more feasible. It is recommended for high-speed token generation.

2. BF16 for Training Stability 3:56

   While NVFP4 is ideal for inference, BF16 or full precision checkpoints should be used if the developer plans to customize the model, fine-tune weights, or use custom quantization algorithms, as this ensures better training stability.

3. Minimal Accuracy Loss 2:30

   Despite the quantization from BF16 to NVFP4, the accuracy degradation is reported to be very small (e.g., 99.99% retained), thanks to intrinsic properties of model weights.

4. Broad Hardware Compatibility 5:40

   The ability to run NVFP4 checkpoints is no longer limited to the newest hardware (Blackwell); it is now supported across a wide range of GPUs, including Hopper and Ampere.

• NVIDIA Developer
• Quantization
• Model Compression
• Inference Optimization
• GPU Architecture
• Large Language Models
• NVFP4 What Is It?
• NVFP4 vs BF16 Notebook

Watch on YouTube Full article

Aug 7, 2026 · 46:01

Compression at the Edge — Chris Alexiuk, NVIDIA

This panel discusses model compression techniques—primarily quantization—that enable running massive Large Language Models (LLMs) on resource-constrained edge devices. Key advancements include formats like NVFP4 and the ability to shrink models dramatically (e.g., GLM 5.2 from 1.5 TB to 250 GB). The discussion emphasizes that successful compression requires understanding model architecture, using advanced methods like Quantization Aware Distillation (QAD), and prioritizing evaluation metrics such as KL divergence over simple accuracy scores.

Key takeaways

1. Model Compression is Critical for Edge AI

   Compression techniques are essential to democratize LLMs, making them viable for local deployment on consumer hardware (e.g., laptops/phones). The goal is enabling powerful models to run without constant reliance on cloud APIs.

2. Advanced Quantization Formats and Techniques 0:04

   NVIDIA's NVFP4 is a specialized 4-bit float format where every group of 16 values shares one FP8 scale. For large models (>20B parameters), Post-Training Quantization (PTQ) works well, while smaller models (<20B) require Quantization Aware Distillation (QAD).

3. Evaluation Focus Shifts to Logits and Architecture 0:08

   Verifying model integrity after compression is complex. The Super Weights paper suggests that quantizing even one number can degrade performance by 20%. Therefore, the preferred signal for evaluation is KL divergence between BF16 and quantized output logits, rather than traditional accuracy scores.

• AI Engineer
• Large Language Models (LLMs)
• Quantization
• Model Compression
• Edge Computing
• AI Architecture
• Nvidia model optimizer space
• Super Weights paper

Watch on YouTube Full article