LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices

TL;DR

This paper introduces LRQ, a low-rank weight quantization method for large language models that improves post-training quantization accuracy by reducing parameter count and enhancing generalization, especially on large multitask tasks.

Contribution

LRQ proposes a low-rank weight-scaling matrix approach for post-training quantization, significantly reducing parameters and improving accuracy over prior methods.

Findings

LRQ outperforms previous PTQ methods on large language models.

LRQ maintains high accuracy with 4-bit weight and 8-bit activation quantization.

LRQ demonstrates robustness across various quantization schemes.

Abstract

With the commercialization of large language models (LLMs), weight-activation quantization has emerged to compress and accelerate LLMs, achieving high throughput while reducing inference costs. However, existing post-training quantization (PTQ) techniques for quantizing weights and activations of LLMs still suffer from non-negligible accuracy drops, especially on massive multitask language understanding. To address this issue, we propose Low-Rank Quantization (LRQ) - a simple yet effective post-training weight quantization method for LLMs that reconstructs the outputs of an intermediate Transformer block by leveraging low-rank weight-scaling matrices, replacing the conventional full weight-scaling matrices that entail as many learnable scales as their associated weights. Thanks to parameter sharing via low-rank structure, LRQ only needs to learn significantly fewer parameters while enabling the individual scaling of weights, thus boosting the generalization capability of quantized LLMs. We show the superiority of LRQ over prior LLM PTQ works under (i) 8-bit weight and per-tensor activation quantization, (ii) 4-bit weight and 8-bit per-token activation quantization, and (iii) low-bit weight-only quantization schemes. Our code is available at Software.