Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

TL;DR

SALR introduces a sparsity-aware low-rank fine-tuning method for large language models, combining pruning and low-rank adaptation to reduce model size and improve inference speed while maintaining performance.

Contribution

The paper proposes SALR, a novel framework that unifies sparse pruning with low-rank adaptation, providing theoretical guarantees and practical efficiency improvements for fine-tuning large language models.

Findings

Achieves 50% sparsity while matching LoRA performance on benchmarks.

Reduces model size by 2 times and speeds up inference by 1.7 times.

Provides theoretical analysis on pruning error bounds and residual information recovery.

Abstract

Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments. Low-rank Adaptation (LoRA) reduces trainable parameters by factorizing weight updates, yet the underlying dense weights still impose high storage and computation costs. Magnitude-based pruning can yield sparse models but typically degrades LoRA's performance when applied naively. In this paper, we introduce SALR (Sparsity-Aware Low-Rank Representation), a novel fine-tuning paradigm that unifies low-rank adaptation with sparse pruning under a rigorous mean-squared-error framework. We prove that statically pruning only the frozen base weights minimizes the pruning error bound, and we recover the discarded residual information via a truncated-SVD low-rank adapter, which provably reduces per-entry MSE by a factor of $(1 - r/\min(d,k))$. To maximize hardware efficiency, we fuse multiple low-rank adapters into a single concatenated GEMM, and we adopt a bitmap-based encoding with a two-stage pipelined decoding + GEMM design to achieve true model compression and speedup. Empirically, SALR attains 50\% sparsity on various LLMs while matching the performance of LoRA on GSM8K and MMLU, reduces model size by $2\times$, and delivers up to a $1.7\times$ inference speedup.