HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs

TL;DR

HASSLE-free is a unified framework for decomposing large language model weights into sparse and low-rank matrices, enabling more efficient compression and inference acceleration while maintaining performance.

Contribution

It introduces a novel optimization framework for sparse plus low-rank decomposition, outperforming prior methods in efficiency and accuracy for large language models.

Findings

Reduces test perplexity by 12% on WikiText-2

Decreases zero-shot task gap by 15%

Outperforms state-of-the-art in decomposition quality

Abstract

The impressive capabilities of large foundation models come at a cost of substantial computing resources to serve them. Compressing these pre-trained models is of practical interest as it can democratize deploying them to the machine learning community at large by lowering the costs associated with inference. A promising compression scheme is to decompose foundation models' dense weights into a sum of sparse plus low-rank matrices. In this paper, we design a unified framework coined HASSLE-free for (semi-structured) sparse plus low-rank matrix decomposition of foundation models. Our framework introduces the local layer-wise reconstruction error objective for this decomposition, we demonstrate that prior work solves a relaxation of this optimization problem; and we provide efficient and scalable methods to minimize the exact introduced optimization problem. HASSLE-free substantially outperforms state-of-the-art methods in terms of the introduced objective and a wide range of LLM evaluation benchmarks. For the Llama3-8B model with a 2:4 sparsity component plus a 64-rank component decomposition, a compression scheme for which recent work shows important inference acceleration on GPUs, HASSLE-free reduces the test perplexity by 12% for the WikiText-2 dataset and reduces the gap (compared to the dense model) of the average of eight popular zero-shot tasks by 15% compared to existing methods.