# BF16 vs NVFP4 with Nemotron 3.5 Lightning

## Executive summary

This technical deep dive compares two model quantization formats, BF16 and NVFP4, using Nemotron 3.5 Lightning. NVFP4 is presented as a highly efficient, low-precision format that significantly reduces memory footprint and increases throughput for inference. While NVFP4 is recommended for token generation, BF16 or full precision checkpoints are advised for model customization, fine-tuning, or training stability. The technology has expanded, allowing NVFP4 to run efficiently on architectures like Hopper and Ampere, not just Blackwell.

## Key takeaways

- NVFP4 for Inference Efficiency: Using NVFP4 drastically reduces the memory footprint required to store model weights compared to BF16, making deployment on smaller hardware more feasible. It is recommended for high-speed token generation.
- BF16 for Training Stability: While NVFP4 is ideal for inference, BF16 or full precision checkpoints should be used if the developer plans to customize the model, fine-tune weights, or use custom quantization algorithms, as this ensures better training stability.
- Minimal Accuracy Loss: Despite the quantization from BF16 to NVFP4, the accuracy degradation is reported to be very small (e.g., 99.99% retained), thanks to intrinsic properties of model weights.
- Broad Hardware Compatibility: The ability to run NVFP4 checkpoints is no longer limited to the newest hardware (Blackwell); it is now supported across a wide range of GPUs, including Hopper and Ampere.

## Technical details

- Quantization Formats: NVFP4 is a low-precision format designed to store and compute numbers (weights) efficiently, similar to INT-4 and FP8. It allows models to fit on smaller hardware while maintaining high performance.
- Performance Metrics: The comparison evaluates memory footprint, throughput (tokens per second), and accuracy across BF16 and NVFP4 variants of Nemotron 3.5 Lightning.
- Model Deployment: The comparison is demonstrated using a notebook that runs measurements on a single H100 GPU, allowing users to reproduce the results.

## Practical implications

- Build Engineers can optimize deployment targets by selecting NVFP4 for inference-heavy services, significantly reducing required GPU RAM and increasing throughput.
- The availability of NVFP4 across older architectures (Hopper, Ampere) simplifies hardware planning and deployment strategy.
- The clear distinction between inference-optimized formats (NVFP4) and training-optimized formats (BF16) guides the development lifecycle, preventing quantization errors during customization.
- The provided notebook allows for direct, reproducible benchmarking of memory and performance gains.

## Topics

Quantization, Model Compression, Inference Optimization, GPU Architecture, Large Language Models, NVFP4 What Is It?, NVFP4 vs BF16 Notebook

Source: https://www.youtube.com/watch?v=Mcwl--BdH3U