LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning

TL;DR

LQ-LoRA introduces a memory-efficient method for fine-tuning large language models by decomposing weight matrices into low-rank and quantized parts, enabling aggressive quantization with minimal performance loss.

Contribution

The paper presents a novel low-rank plus quantized matrix decomposition technique for efficient language model fine-tuning, outperforming existing quantization baselines and enabling sub-3-bit quantization.

Findings

Outperforms QLoRA and GPTQ-LoRA baselines.

Enables quantization to below 3 bits with minor performance loss.

Achieves effective model compression with 2.75-bit LLaMA-2-70B.

Abstract

We propose a simple approach for memory-efficient adaptation of pretrained language models. Our approach uses an iterative algorithm to decompose each pretrained matrix into a high-precision low-rank component and a memory-efficient quantized component. During finetuning, the quantized component remains fixed and only the low-rank component is updated. We present an integer linear programming formulation of the quantization component which enables dynamic configuration of quantization parameters (e.g., bit-width, block size) for each matrix given an overall target memory budget. We further explore a data-aware version of the algorithm which uses an approximation of the Fisher information matrix to weight the reconstruction objective during matrix decomposition. Experiments on finetuning RoBERTa and LLaMA-2 (7B and 70B) demonstrate that our low-rank plus quantized matrix decomposition approach (LQ-LoRA) outperforms strong QLoRA and GPTQ-LoRA baselines and enables aggressive quantization to sub-3 bits with only minor performance degradations. When finetuned on a language modeling calibration dataset, LQ-LoRA can also be used for model compression; in this setting our 2.75-bit LLaMA-2-70B model (which has 2.85 bits on average when including the low-rank components and requires 27GB of GPU memory) performs respectably compared to the 16-bit baseline.