Hierarchical Sparse Plus Low Rank Compression of LLM

TL;DR

This paper introduces Hierarchical Sparse Plus Low-Rank (HSS) compression for large language models, combining sparsity and low-rank factorization to reduce memory and computation while maintaining performance.

Contribution

The paper proposes a novel two-stage HSS compression method with recursive hierarchy and RCM permutation, improving efficiency and compressibility of LLMs.

Findings

HSS achieves significant memory savings on LLaMA-7B.

HSS maintains state-of-the-art perplexity scores.

HSS outperforms classical sparse-plus-SVD methods.

Abstract

Modern large language models (LLMs) place extraordinary pressure on memory and compute budgets, making principled compression indispensable for both deployment and continued training. We present Hierarchical Sparse Plus Low-Rank (HSS) compression, a two-stage scheme that (i) removes the largest-magnitude weights into a sparse matrix S and (ii) applies a recursive Hierarchically Sparse Separable (HSS) low-rank factorisation to the dense residual matrix. A recursive rank-reducing strategy and a reverse Cuthill-Mckee (RCM) permutation are introduced to align high weights towards the diagonal with the block-diagonal hierarchy, maximising off-diagonal compressibility (because they are touched only once). HSS is hardware-friendly: its matrix-vector multiply reduces to one sparse and a sequence of thin-matrix multiplications and can be trained end-to-end with standard optimisers. Experiments on LLaMA-7B show that targeting only the self-attention projections (1.6 B parameters of Q, K, and V matrices out of a total 7B parameters) suffices to yield large memory savings while retaining comparable state-of-the-art perplexity scores on test samples of the WikiText dataset. For example, with a 30\% sparsity budget and an outer rank of 512, sHSS-RCM achieves a perplexity of 1.64, outperforming dense baselines and classical sparse-plus-SVD variants, while also achieving significant memory savings.