Low-Rank Compression of Language Models via Differentiable Rank Selection

TL;DR

This paper introduces LLRC, a gradient-based, fine-tuning-free method for selecting optimal low-rank decompositions in large language models, improving compression efficiency and task performance without additional training.

Contribution

We propose LLRC, a novel differentiable rank selection method that optimizes singular value masks directly, outperforming existing heuristics and gradient-based approaches in model compression tasks.

Findings

LLRC outperforms heuristic and gradient-based methods across multiple tasks.

Our approach achieves higher accuracy at 20% compression rate on Llama-2-13B.

LLRC performs competitively with fine-tuning methods without additional training.

Abstract

Approaches for compressing large-language models using low-rank decomposition have made strides, particularly with the introduction of activation and loss-aware SVD, which improves the trade-off between decomposition rank and downstream task performance. Despite these advancements, a persistent challenge remains--selecting the optimal ranks for each layer to jointly optimise compression rate and downstream task accuracy. Current methods either rely on heuristics that can yield sub-optimal results due to their limited discrete search space or are gradient-based but are not as performant as heuristic approaches without post-compression fine-tuning. To address these issues, we propose Learning to Low-Rank Compress (LLRC), a gradient-based approach which directly learns the weights of masks that select singular values in a fine-tuning-free setting. Using a calibration dataset, we train only the mask weights to select fewer and fewer singular values while minimising the divergence of intermediate activations from the original model. Our approach outperforms competing ranking selection methods that similarly require no post-compression fine-tuning across various compression rates on common-sense reasoning and open-domain question-answering tasks. For instance, with a compression rate of 20% on Llama-2-13B, LLRC outperforms the competitive Sensitivity-based Truncation Rank Searching (STRS) on MMLU, BoolQ, and OpenbookQA by 12%, 3.5%, and 4.4%, respectively. Compared to other compression techniques, our approach consistently outperforms fine-tuning-free variants of SVD-LLM and LLM-Pruner across datasets and compression rates. Our fine-tuning-free approach also performs competitively with the fine-tuning variant of LLM-Pruner.