DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs

TL;DR

DuQuant introduces a dual transformation approach using rotation and permutation to effectively mitigate both normal and massive outliers in LLM quantization, significantly improving low-bit model performance.

Contribution

It proposes a novel dual transformation method with rotation and permutation to better handle outliers in LLM quantization, outperforming existing methods.

Findings

Outperforms state-of-the-art baselines across various LLMs and tasks.

Effectively manages both normal and massive outliers in low-bit quantization.

Enhances model performance with 4-bit weight-activation quantization.

Abstract

Quantization of large language models (LLMs) faces significant challenges, particularly due to the presence of outlier activations that impede efficient low-bit representation. Traditional approaches predominantly address Normal Outliers, which are activations across all tokens with relatively large magnitudes. However, these methods struggle with smoothing Massive Outliers that display significantly larger values, which leads to significant performance degradation in low-bit quantization. In this paper, we introduce DuQuant, a novel approach that utilizes rotation and permutation transformations to more effectively mitigate both massive and normal outliers. First, DuQuant starts by constructing the rotation matrix, using specific outlier dimensions as prior knowledge, to redistribute outliers to adjacent channels by block-wise rotation. Second, We further employ a zigzag permutation to balance the distribution of outliers across blocks, thereby reducing block-wise variance. A subsequent rotation further smooths the activation landscape, enhancing model performance. DuQuant simplifies the quantization process and excels in managing outliers, outperforming the state-of-the-art baselines across various sizes and types of LLMs on multiple tasks, even with 4-bit weight-activation quantization. Our code is available at https://github.com/Hsu1023/DuQuant.