IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuning

TL;DR

IncreLoRA is a novel incremental parameter allocation method for PEFT that adaptively adds trainable parameters based on module importance, improving efficiency especially in low-resource scenarios.

Contribution

It introduces an adaptive parameter addition approach for LoRA, overcoming preset limitations and enhancing parameter efficiency during fine-tuning.

Findings

Outperforms baselines in low-resource settings on GLUE.

Achieves higher parameter efficiency with comparable or better performance.

Effectively allocates parameters based on module importance scores.

Abstract

With the increasing size of pre-trained language models (PLMs), fine-tuning all the parameters in the model is not efficient, especially when there are a large number of downstream tasks, which incur significant training and storage costs. Many parameter-efficient fine-tuning (PEFT) approaches have been proposed, among which, Low-Rank Adaptation (LoRA) is a representative approach that injects trainable rank decomposition matrices into every target module. Yet LoRA ignores the importance of parameters in different modules. To address this problem, many works have been proposed to prune the parameters of LoRA. However, under limited training conditions, the upper bound of the rank of the pruned parameter matrix is still affected by the preset values. We, therefore, propose IncreLoRA, an incremental parameter allocation method that adaptively adds trainable parameters during training based on the importance scores of each module. This approach is different from the pruning method as it is not limited by the initial number of training parameters, and each parameter matrix has a higher rank upper bound for the same training overhead. We conduct extensive experiments on GLUE to demonstrate the effectiveness of IncreLoRA. The results show that our method owns higher parameter efficiency, especially when under the low-resource settings where our method significantly outperforms the baselines. Our code is publicly available.