Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization

TL;DR

This paper introduces Noise Perturbation Fine-tuning (NPFT), a method to improve quantization of large language models by reducing outlier sensitivity, enabling efficient low-precision deployment without special outlier handling.

Contribution

The paper proposes NPFT, a novel fine-tuning approach that reduces outlier sensitivity in LLM weights, improving quantization performance and inference efficiency.

Findings

NPFT improves quantization performance on OPT and LLaMA models.

NPFT achieves performance comparable to GPTQ on LLaMA2-7B-4bits.

NPFT enhances inference efficiency with stable results across quantizers.

Abstract

Quantization is a critical step to enable efficient LLM serving under limited resource. However, previous research observes that certain weights in the LLM, known as outliers, are significantly sensitive to quantization noises. Existing quantization methods leave these outliers as floating points or higher precisions to retain performance, posting challenges on the efficient hardware deployment of the mixed-precision model. This work investigates an alternative way to tame the sensitive weights' impact on the quantization error, by reducing the loss Hessian trace with respect to outliers through an efficient fine-tuning process. We propose Noise Perturbation Fine-tuning (NPFT), which identifies outlier weights and add random weight perturbations on the outliers as the model going through a PEFT optimization. NPFT tames the sensitivity of outlier weights so that the quantized model performance can be improved without special treatment to the outliers. When applied to OPT and LLaMA models, our NPFT method achieves stable performance improvements for both uniform and non-uniform quantizers, while also offering better inference efficiency. Notably, the simplest RTN can achieve performance on par with GPTQ using our NPFT on LLaMA2-7B-4bits benchmark.