OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization

TL;DR

OSAQ introduces a novel low-rank Hessian-based method for suppressing weight outliers in LLMs, significantly improving low-bit quantization performance without additional inference costs.

Contribution

The paper proposes a second-order null space approach for outlier suppression in LLM quantization, enabling efficient, additive weight transformations with no extra inference overhead.

Findings

OSAQ achieves over 40% lower perplexity in 2-bit quantization with GPTQ.

The method effectively suppresses weight outliers and improves quantization accuracy.

It operates efficiently using a closed-form solution without iterative training.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities. However, their massive parameter scale leads to significant resource consumption and latency during inference. Post-training weight-only quantization offers a promising solution by reducing model size and accelerating token generation through alleviating the memory-bound issue. Nevertheless, the presence of inherent systematic outliers in weights continues to be a major obstacle. While existing methods, such as scaling and rotation, attempt to address this issue, the performance remains unsatisfactory. In this paper, we propose Outlier Self-Absorption Quantization (OSAQ), which performs additive weight suppression guided by the second-order low-rank property for low-bit weight-only quantization of LLMs. Specifically, we observe that the Hessian exhibits low-rank consistency across different inputs, with certain directions consistently showing vanishing curvature. Leveraging this property, we identify a stable null space of the Hessian and then construct an additive weight transformation by linearly combining the vectors within this null space, thereby suppressing weight outliers without affecting the task loss. This additive transformation can be absorbed into the weights offline, requiring no inter-layer transformations and introducing no inference overhead. Moreover, the construction is efficiently achieved by a closed-form solution, without resource-intensive training or iterative procedures. Extensive experiments demonstrate that OSAQ effectively suppresses outliers and enhances low-bit quantization performance. For instance, in 2-bit quantization, OSAQ, when integrated with GPTQ, achieves over 40% lower perplexity compared to vanilla GPTQ.