RILQ: Rank-Insensitive LoRA-based Quantization Error Compensation for Boosting 2-bit Large Language Model Accuracy

TL;DR

RILQ introduces a rank-insensitive method for quantization error compensation, significantly improving 2-bit LLM accuracy while maintaining efficiency, addressing limitations of previous approaches in ultra-low-bit scenarios.

Contribution

The paper proposes RILQ, a novel rank-insensitive error compensation technique that enhances 2-bit LLM performance, filling a gap in quantization error correction for ultra-low-bit models.

Findings

Consistent accuracy improvements on LLaMA-2 and LLaMA-3.

Effective across various state-of-the-art quantizers.

Maintains computational efficiency comparable to existing methods.

Abstract

Low-rank adaptation (LoRA) has become the dominant method for parameter-efficient LLM fine-tuning, with LoRA-based quantization error compensation (LQEC) emerging as a powerful tool for recovering accuracy in compressed LLMs. However, LQEC has underperformed in sub-4-bit scenarios, with no prior investigation into understanding this limitation. We propose RILQ (Rank-Insensitive LoRA-based Quantization Error Compensation) to understand fundamental limitation and boost 2-bit LLM accuracy. Based on rank analysis revealing model-wise activation discrepancy loss's rank-insensitive nature, RILQ employs this loss to adjust adapters cooperatively across layers, enabling robust error compensation with low-rank adapters. Evaluations on LLaMA-2 and LLaMA-3 demonstrate RILQ's consistent improvements in 2-bit quantized inference across various state-of-the-art quantizers and enhanced accuracy in task-specific fine-tuning. RILQ maintains computational efficiency comparable to existing LoRA methods, enabling adapter-merged weight-quantized LLM inference with significantly enhanced accuracy, making it a promising approach for boosting 2-bit LLM performance. Our code is available at https://github.com/aiha-lab/RILQ.