OAC: Output-adaptive Calibration for Accurate Post-training Quantization

TL;DR

This paper introduces Output-adaptive Calibration (OAC), a novel post-training quantization method for large language models that incorporates output information to improve accuracy at low bit-widths.

Contribution

OAC formulates a new output-aware quantization error and approximates an output-adaptive Hessian to enhance low-precision LLM quantization accuracy.

Findings

OAC outperforms state-of-the-art methods like SpQR and BiLLM.

OAC achieves better accuracy at 2-bit and binary quantization.

The method effectively maintains model output quality after quantization.

Abstract

Deployment of Large Language Models (LLMs) has major computational costs, due to their rapidly expanding size. Compression of LLMs reduces the memory footprint, latency, and energy required for their inference. Post-training Quantization (PTQ) techniques have been developed to compress LLMs while avoiding expensive re-training. Most PTQ approaches formulate the quantization error based on a layer-wise Euclidean loss, ignoring the model output. Then, each layer is calibrated using its layer-wise Hessian to update the weights towards minimizing the quantization error. The Hessian is also used for detecting the most salient weights to quantization. Such PTQ approaches are prone to accuracy drop in low-precision quantization. We propose Output-adaptive Calibration (OAC) to incorporate the model output in the calibration process. We formulate the quantization error based on the distortion of the output cross-entropy loss. OAC approximates the output-adaptive Hessian for each layer under reasonable assumptions to reduce the computational complexity. The output-adaptive Hessians are used to update the weight matrices and detect the salient weights towards maintaining the model output. Our proposed method outperforms the state-of-the-art baselines such as SpQR and BiLLM, especially, at extreme low-precision (2-bit and binary) quantization.