# GSQ-RCO: Better Non-Uniform GGUF Quantization

## Executive summary

This talk introduces GSQ (Gumbel Softmax Quantization) and RCO (Riemannian Optimization), two complementary methods designed to achieve highly accurate low-bit quantization of Large Language Models (LLMs). GSQ improves the quality of scalar quantization (K-quant) by utilizing stochastic optimization, allowing backpropagation through quantization bins. RCO addresses the complex problem of non-uniform bit allocation and expert pruning by optimizing model parameters under strict, multiple size budget constraints. Together, these methods enable state-of-the-art performance in the ultra-low bit regime (e.g., 2.5 bits), significantly improving model efficiency for deployment.

## Key takeaways

- GSQ Closes the Quantization Gap: GSQ addresses the observed performance gap between simple scalar quantization methods (like GPTQ) and vector quantization formats (like Q-Tip) when moving below 4 bits. It achieves this by using Gumbel-Softmax sampling to enable stochastic optimization, allowing the model to explore the discrete quantization space while maintaining differentiability for gradient updates. (See 1:20:00)
- RCO Solves Constrained Bit Allocation: RCO is a general optimization primitive for allocating bit widths (mixed precision quantization) or pruning experts (MoE models) under multiple, fixed size budgets. It solves the difficult problem of non-decomposable objectives, where the loss depends on all chosen parameters simultaneously. (See 1:25:00)
- High Efficiency and Scalability: The combined GSQ-RCO pipeline allows for efficient quantization, including parallelizing the process across experts in MoE models. Benchmarks show significant gains, such as a 10% increase in live code bench performance (from 60 to almost 70) on the Chinchilla 2.5 model. (See 1:30:00)

## Technical details

- Quantization Fundamentals: Quantization compresses LLM weights (which can be massive, e.g., 140 GB for a 70B model) from high precision (e.g., FP16) down to low bits (e.g., 2-4 bits). Formats include GGUF (compatible with GGML), K-quant (per element), and I-quant (group quantization). (See 0:02:00)
- GSQ Mechanism: GSQ uses Gumbel-Softmax sampling to treat the quantization process as a stochastic optimization problem. This allows the model to sample from the quantization grid and backpropagate gradients through the discrete bins, which is crucial for optimizing the weights. The process is stable using the Lion optimizer, which helps maintain gradient signal when logits become weak. (See 0:02:40)
- RCO Mechanism: RCO optimizes model parameters by treating the problem as an optimization on a Riemannian manifold. This allows the optimizer to move only in directions that maintain the overall size or budget (e.g., keeping the total bit width fixed at 3.5 bits), solving the non-decomposable nature of the objective. (See 0:25:00)

## Practical implications

- Enables deployment of massive LLMs (e.g., 1T parameter models) with significantly reduced memory footprint and improved inference throughput.
- The modular nature of GSQ and RCO allows build engineers to optimize different aspects of the model (e.g., layer-wise quantization vs. expert pruning) independently.
- The ability to parallelize quantization across experts in MoE models simplifies scaling the quantization process on large GPU clusters.

## Topics

Large Language Models (LLMs), Quantization, Stochastic Optimization, Model Compression, Riemannian Geometry, Build Engineering, GSQ-RCO GGUF collection, Qwen3.8-27B GSQ-RCO GGUF

Source: https://www.youtube.com/watch?v=ECZPAzT6VgQ