ReALLM: A general framework for LLM compression and fine-tuning

TL;DR

ReALLM is a versatile framework that compresses and fine-tunes large language models efficiently using low-bit quantization and a novel matrix decomposition approach, enabling high performance with minimal memory usage.

Contribution

ReALLM introduces a unified method combining post-training quantization and fine-tuning for language models using less than 4 bits per parameter, with adaptive matrix representations and a neural decoder.

Findings

Achieves state-of-the-art results at 2-bit quantization after fine-tuning.

Outperforms existing methods on language generation benchmarks.

Requires only one forward pass for matrix decompression.

Abstract

We introduce ReALLM, a novel approach for compression and memory-efficient adaptation of pre-trained language models that encompasses most of the post-training quantization and fine-tuning methods for a budget of <4 bits. Pre-trained matrices are decomposed into a high-precision low-rank component and a vector-quantized latent representation (using an autoencoder). During the fine-tuning step, only the low-rank components are updated. Our results show that pre-trained matrices exhibit different patterns. ReALLM adapts the shape of the encoder (small/large embedding, high/low bit VQ, etc.) to each matrix. ReALLM proposes to represent each matrix with a small embedding on $b$ bits and a neural decoder model $\mathcal{D}_\phi$ with its weights on $b_\phi$ bits. The decompression of a matrix requires only one embedding and a single forward pass with the decoder. Our weight-only quantization algorithm yields the best results on language generation tasks (C4 and WikiText-2) for a budget of $3$ bits without any training. With a budget of $2$ bits, ReALLM achieves state-of-the art performance after fine-tuning on a small calibration dataset.