LoPRo: Enhancing Low-Rank Quantization via Permuted Block-Wise Rotation

TL;DR

LoPRo introduces a fine-tuning-free low-rank quantization method that uses block-wise permutation and Walsh-Hadamard transforms to improve accuracy at 2-3 bits, outperforming existing methods on large language models.

Contribution

The paper presents LoPRo, a novel low-rank quantization approach that enhances residual matrix quantization with permutation and transformation techniques, eliminating the need for fine-tuning.

Findings

Outperforms existing fine-tuning-free PTQ methods at 2- and 3-bit quantization.

Achieves state-of-the-art accuracy on LLaMA-2 and LLaMA-3 models.

Reduces perplexity and improves accuracy on Mixtral-8x7B with efficient quantization.

Abstract

Post-training quantization (PTQ) enables effective model compression while preserving relatively high accuracy. Current weight-only PTQ methods primarily focus on the challenging sub-3-bit regime, where approaches often suffer significant accuracy degradation, typically requiring fine-tuning to achieve competitive performance. In this work, we revisit the fundamental characteristics of weight quantization and analyze the challenges in quantizing the residual matrix under low-rank approximation. We propose LoPRo, a novel fine-tuning-free PTQ algorithm that enhances residual matrix quantization by applying block-wise permutation and Walsh-Hadamard transformations to rotate columns of similar importance, while explicitly preserving the quantization accuracy of the most salient column blocks. Furthermore, we introduce a mixed-precision fast low-rank decomposition based on rank-1 sketch (R1SVD) to further minimize quantization costs. Experiments demonstrate that LoPRo outperforms existing fine-tuning-free PTQ methods at both 2-bit and 3-bit quantization, achieving accuracy comparable to fine-tuning baselines. Specifically, LoPRo achieves state-of-the-art quantization accuracy on LLaMA-2 and LLaMA-3 series models while delivering up to a 4$\times$ speedup. In the MoE model Mixtral-8x7B, LoPRo completes quantization within 2.5 hours, simultaneously reducing perplexity by 0.4$\downarrow$ and improving accuracy by 8\%$\uparrow$. Moreover, compared to other low-rank quantization methods, LoPRo achieves superior accuracy with a significantly lower rank, while maintaining high inference efficiency and minimal additional latency.