WeightLoRA: Keep Only Necessary Adapters

TL;DR

WeightLoRA adaptively selects the most critical adapters during training, significantly reducing parameters needed while maintaining or improving performance across various language models.

Contribution

This paper introduces WeightLoRA, a novel adaptive method for selecting LoRA adapters, reducing memory usage and simplifying layer selection in parameter-efficient fine-tuning.

Findings

WeightLoRA reduces trainable parameters significantly.

WeightLoRA maintains or improves performance on benchmarks.

WeightLoRA+ outperforms other adaptive methods in most cases.

Abstract

The widespread utilization of language models in modern applications is inconceivable without Parameter-Efficient Fine-Tuning techniques, such as low-rank adaptation ($\texttt{LoRA}$), which adds trainable adapters to selected layers. Although $\texttt{LoRA}$ may obtain accurate solutions, it requires significant memory to train large models and intuition on which layers to add adapters. In this paper, we propose a novel method, $\texttt{WeightLoRA}$, which overcomes this issue by adaptive selection of the most critical $\texttt{LoRA}$ heads throughout the optimization process. As a result, we can significantly reduce the number of trainable parameters while maintaining the capability to obtain consistent or even superior metric values. We conduct experiments for a series of competitive benchmarks and DeBERTa, BART, and Llama models, comparing our method with different adaptive approaches. The experimental results demonstrate the efficacy of $\texttt{WeightLoRA}$ and the superior performance of $\texttt{WeightLoRA+}$ in almost all cases.