DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

TL;DR

DropLoRA introduces a pruning-based method that dynamically adapts the low-rank subspace in PEFT, significantly improving performance over traditional LoRA in various large language model tasks without extra costs.

Contribution

It proposes DropLoRA, a novel dynamic low-rank adaptation method that overcomes the static subspace limitation of LoRA, enhancing fine-tuning performance.

Findings

DropLoRA outperforms LoRA across multiple tasks.

It improves performance without additional training costs.

Effective on LLaMA models for diverse NLP tasks.

Abstract

LoRA-based large model parameter-efficient fine-tuning (PEFT) methods use low-rank de- composition to approximate updates to model parameters. However, compared to full- parameter fine-tuning, low-rank updates often lead to a performance gap in downstream tasks. To address this, we introduce DropLoRA, a novel pruning-based approach that focuses on pruning the rank dimension. Unlike conven- tional methods that attempt to overcome the low-rank bottleneck, DropLoRA innovatively integrates a pruning module between the two low-rank matrices in LoRA to simulate dy- namic subspace learning. This dynamic low- rank subspace learning allows DropLoRA to overcome the limitations of traditional LoRA, which operates within a static subspace. By continuously adapting the learning subspace, DropLoRA significantly boosts performance without incurring additional training or infer- ence costs. Our experimental results demon- strate that DropLoRA consistently outperforms LoRA in fine-tuning the LLaMA series across a wide range of large language model gener- ation tasks, including commonsense reason- ing, mathematical reasoning, code generation, and instruction-following. Our code is avail- able at https://github.com/TayeeChang/DropLoRA.