ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization

TL;DR

This paper introduces ASER, a novel quantization algorithm for large language models that effectively reduces errors and preserves accuracy in low-bit settings through activation smoothing and error reconstruction techniques.

Contribution

ASER combines low-rank error reconstruction and activation smoothing to improve low-bit quantization of LLMs, addressing key challenges in model compression.

Findings

ASER achieves competitive accuracy in W4A8 quantization.

It effectively reduces quantization errors with minor computational overhead.

ASER demonstrates potential for activation quantization in LLMs.

Abstract

Quantization stands as a pivotal technique for large language model (LLM) serving, yet it poses significant challenges particularly in achieving effective low-bit quantization. The limited numerical mapping makes the quantized model produce a non-trivial error, bringing out intolerable performance degration. This paper is anchored in the basic idea of model compression objectives, and delves into the layer-wise error distribution of LLMs during post-training quantization. Subsequently, we introduce ASER, an algorithm consisting of (1) Error Reconstruction: low-rank compensation for quantization error with LoRA-style matrices constructed by whitening SVD; (2) Activation Smoothing: outlier extraction to gain smooth activation and better error compensation. ASER is capable of quantizing typical LLMs to low-bit ones, particularly preserving accuracy even in W4A8 per-channel setup. Experimental results show that ASER is competitive among the state-of-the-art quantization algorithms, showing potential to activation quantization, with minor overhead.