LQER: Low-Rank Quantization Error Reconstruction for LLMs

TL;DR

LQER introduces a novel low-rank quantization error reduction method that significantly improves post-training quantization of large language models, achieving near-lossless performance with reduced hardware resources without complex optimization procedures.

Contribution

The paper proposes LQER, a new approach combining quantization and low-rank approximation that simplifies the process and enhances performance of LLM quantization without requiring iterative optimization.

Findings

Achieves near-lossless W4A8 quantization on various LLMs.

Reduces hardware resource usage by 1.36 times compared to state-of-the-art.

Eliminates the need for specialized memory collection processes.

Abstract

Post-training quantization of Large Language Models (LLMs) is challenging. In this work, we introduce Low-rank Quantization Error Reduction (LQER), which combines quantization and low-rank approximation to recover the model capability. LQER leverages an activation-induced scale matrix to drive the singular value distribution of quantization error towards a desirable distribution, which enables nearly-lossless W4A8 quantization on various LLMs and downstream tasks without the need for knowledge distillation, grid search, or gradient-base iterative optimization. Unlike existing methods, the computation pattern of LQER eliminates the need for specialized Scatter and Gather processes to collect high-precision weights from irregular memory locations. Our W4A8 LLMs achieve near-lossless performance on six popular downstream tasks, while using 1.36$\times$ fewer hardware resources than the leading state-of-the-art method. We open-source our framework at https://github.com/ChengZhang-98/lqer