# Lecture 118: GSQ-RCO: Better Non-Uniform GGUF Quantization

## Executive summary

This talk introduces GSQ (Gumbel Softmax Quantization) and RCO (Riemannian optimization for bit allocation), two complementary methods for achieving highly accurate, low-bit quantization of Large Language Models (LLMs). GSQ addresses the limitations of scalar quantization below 4 bits by using stochastic optimization and Gumbel Softmax sampling to maintain quantization fidelity. RCO provides a smart mechanism for non-uniform bit allocation across a model, recognizing that different layers (e.g., K/V vs. down projection) have varying sensitivities. Together, these methods enable state-of-the-art performance in ultra-low bit regimes (e.g., 2.5 bits) while maintaining compatibility with the GGUF format, significantly improving model size and throughput.

## Key takeaways

- GSQ for Low-Bit Quantization: GSQ tackles the quantization gap observed when moving below 4 bits by treating quantization as a stochastic optimization problem. It uses Gumbel Softmax sampling to estimate the optimal quantization point, allowing backpropagation through the discrete quantization bins. This approach is crucial for maintaining accuracy in ultra-low bit regimes, approaching the performance of advanced formats like QTiP.
- RCO for Non-Uniform Bit Allocation: RCO solves the problem of optimally allocating a fixed bit budget across a model's layers. Since different layers have varying sensitivities (e.g., K/V layers vs. down projection), RCO uses Riemannian optimization to find the best bit-width assignment, which is critical for achieving maximum compression efficiency.
- System Integration and Performance: The combination of GSQ and RCO allows for efficient quantization of massive models (e.g., Kimi K2.5, 1T model) into the GGUF format. The methods support parallelization, enabling large-scale quantization across multiple GPUs, and achieving significant throughput gains (up to 6.2x BF16 throughput).

## Technical details

- Quantization Formats and Challenges: Quantization is a discrete optimization problem aiming to find the closest low-bit model. While conventional methods include scalar quantization (e.g., GPTQ) and vector quantization (e.g., AQLM), the GGUF format supports both K-quant (per element) and IQ-quant (group quantization). The challenge is that simple scalar quantization falls short compared to vector methods at very low bit widths.
- GSQ Mechanism (Gumbel Softmax): GSQ uses Gumbel Softmax sampling to estimate the optimal quantization point by assigning probabilities to quantization grid points. This stochastic approach allows for backpropagation through the discrete quantization bins, which is necessary for gradient-based optimization. The process is computationally intensive, requiring many forward passes for convergence.
- RCO Mechanism (Riemannian Optimization): RCO addresses the non-decomposable nature of model loss when optimizing bit widths. Instead of simple sensitivity measures or slow evolutionary search, RCO uses Riemannian optimization to constrain the search space, ensuring that the allocated bit widths maintain the overall model budget (e.g., 3.5 bits average).
- Optimization and Scaling: The optimization process benefits from using the Lion optimizer, which stabilizes training when gradients become extremely small (a common issue with Softmax). Furthermore, the staged quantization process (e.g., optimizing QKV block, then Attention block, then MLP block) allows for parallel computation and scaling across multiple GPUs.

## Practical implications

- Enables the deployment of massive LLMs (e.g., 1T parameter models) at ultra-low bit widths (down to 2.5 bits) while maintaining high accuracy.
- The GGUF format compatibility ensures that the advanced quantization techniques are accessible across a broad range of consumer and specialized hardware.
- The ability to parallelize the quantization process makes scaling the model compression routine feasible even on large clusters of GPUs.

## Topics

LLM Quantization, GGUF Format, Stochastic Optimization, Riemannian Optimization, Mixture of Experts (MoE), Model Compression, GSQ-RCO GGUF collection, Qwen3.8-27B GSQ-RCO GGUF

Source: https://www.youtube.com/watch?v=ISaPeJeXJnY