LoRA-Squeeze: Simple and Effective Post-Tuning and In-Tuning Compression of LoRA Modules

TL;DR

LoRA-Squeeze enhances parameter-efficient fine-tuning by learning a high-rank solution first and then compressing it via RSVD, resulting in better performance and flexibility across diverse tasks.

Contribution

It introduces a simple, effective post-hoc and in-tuning rank compression method for LoRA modules, improving adaptability and performance.

Findings

Post-hoc compression yields lower-rank adapters that outperform direct training at the target rank.

Gradual in-tuning rank annealing achieves optimal size-performance trade-offs.

Method is effective across multiple text and vision-language tasks.

Abstract

Despite its huge number of variants, standard Low-Rank Adaptation (LoRA) is still a dominant technique for parameter-efficient fine-tuning (PEFT). Nonetheless, it faces persistent challenges, including the pre-selection of an optimal rank and rank-specific hyper-parameters, as well as the deployment complexity of heterogeneous-rank modules and more sophisticated LoRA derivatives. In this work, we introduce LoRA-Squeeze, a simple and efficient methodology that aims to improve standard LoRA learning by changing LoRA module ranks either post-hoc or dynamically during training}. Our approach posits that it is better to first learn an expressive, higher-rank solution and then compress it, rather than learning a constrained, low-rank solution directly. The method involves fine-tuning with a deliberately high(er) source rank, reconstructing or efficiently approximating the reconstruction of the full weight update matrix, and then using Randomized Singular Value Decomposition (RSVD) to create a new, compressed LoRA module at a lower target rank. Extensive experiments across 13 text and 10 vision-language tasks show that post-hoc compression often produces lower-rank adapters that outperform those trained directly at the target rank, especially if a small number of fine-tuning steps at the target rank is allowed. Moreover, a gradual, in-tuning rank annealing variant of LoRA-Squeeze consistently achieves the best LoRA size-performance trade-off.