ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference

TL;DR

ParoQuant introduces a novel pairwise rotation quantization technique that effectively reduces outliers in large language models, improving inference accuracy with minimal overhead, thereby enabling more efficient deployment of reasoning LLMs.

Contribution

The paper proposes ParoQuant, a new PTQ method combining Givens rotations and channel-wise scaling to address outliers and improve accuracy in LLM inference.

Findings

Achieves 2.4% accuracy improvement over AWQ on reasoning tasks.

Maintains less than 10% inference overhead.

Matches state-of-the-art quantization accuracy.

Abstract

Post-training quantization (PTQ) compresses the weights and activations of large language models (LLMs) into low-precision representations to reduce memory footprint and accelerate inference. However, the presence of outliers in weights and activations often leads to large quantization errors and severe accuracy degradation, especially in recent reasoning LLMs where errors accumulate across long chains of thought. Existing PTQ methods either fail to sufficiently suppress outliers or introduce significant overhead during inference. In this paper, we propose Pairwise Rotation Quantization (ParoQuant), a PTQ method that combines hardware-efficient and optimizable independent Givens rotations with channel-wise scaling to even out the magnitudes across channels and narrow the dynamic range within each quantization group, effectively addressing the outlier issue. We further co-design the inference kernel to fully exploit GPU parallelism and keep the rotations and scaling lightweight at runtime. Under weight-only quantization, ParoQuant achieves an average 2.4% accuracy improvement over AWQ on reasoning tasks, with less than 10% overhead. ParoQuant also matches the accuracy of state-of-the-art weight-activation quantization methods. This paves the way for more efficient and accurate deployment of reasoning LLMs.