# Compression at the Edge — Chris Alexiuk, NVIDIA

## Executive summary

This panel discusses model compression techniques—primarily quantization—that enable running massive Large Language Models (LLMs) on resource-constrained edge devices. Key advancements include formats like NVFP4 and the ability to shrink models dramatically (e.g., GLM 5.2 from 1.5 TB to 250 GB). The discussion emphasizes that successful compression requires understanding model architecture, using advanced methods like Quantization Aware Distillation (QAD), and prioritizing evaluation metrics such as KL divergence over simple accuracy scores.

## Key takeaways

- Model Compression is Critical for Edge AI: Compression techniques are essential to democratize LLMs, making them viable for local deployment on consumer hardware (e.g., laptops/phones). The goal is enabling powerful models to run without constant reliance on cloud APIs.
- Advanced Quantization Formats and Techniques: NVIDIA's NVFP4 is a specialized 4-bit float format where every group of 16 values shares one FP8 scale. For large models (>20B parameters), Post-Training Quantization (PTQ) works well, while smaller models (<20B) require Quantization Aware Distillation (QAD).
- Evaluation Focus Shifts to Logits and Architecture: Verifying model integrity after compression is complex. The Super Weights paper suggests that quantizing even one number can degrade performance by 20%. Therefore, the preferred signal for evaluation is KL divergence between BF16 and quantized output logits, rather than traditional accuracy scores.

## Technical details

- Quantization Formats: NVFP4 is a 4-bit float format utilizing microlock scaling, where 16 elements share one FP8 scale. Other formats discussed include BF16 and various bit-depths (e.g., Q1_3.6).
- Compression Methods: Techniques range from simple quantization to advanced methods like Quantization Aware Distillation (QAD) and model routing, which can use large models for planning and small models for execution.
- Model Architecture Sensitivity: Layer importance is highly unequal; the first and last layers often carry enormous weight, while middle layers may be less critical. Specific components like linear attention projection layers and KV QKB layers are noted as being particularly sensitive to quantization.
- Compression Scaling: A model like GLM 5.2 can be shrunk from 1.5 terabytes (TB) down to 250 GB, achieving an 86% reduction in size without a proportional loss of intelligence.

## Practical implications

- Businesses can leverage compression not only for local deployment but also to increase concurrency and reduce compute costs by serving models on individual machines.
- The trend suggests that the most valuable approach is training a massive model and then quantizing it, rather than starting with a small-precision model.
- Developers should consider speed/throughput alongside size, as smaller models (e.g., 200 tokens/second) may be faster than highly compressed large models (e.g., 5-10 tokens/second).

## Topics

Large Language Models (LLMs), Quantization, Model Compression, Edge Computing, AI Architecture, Nvidia model optimizer space, Super Weights paper

Source: https://www.youtube.com/watch?v=J4_jCrTxMkk