Sparsity May Be All You Need: Sparse Random Parameter Adaptation

TL;DR

This paper shows that randomly selecting a small subset of model parameters to fine-tune can be as effective as structured PEFT methods like LoRA, emphasizing the importance of the number of trainable parameters over their specific arrangement.

Contribution

Introducing a simple random parameter selection method for PEFT that challenges the necessity of structured adapters like LoRA, highlighting the role of parameter count in performance.

Findings

Random parameter selection is competitive with LoRA.

Number of trainable parameters is crucial for PEFT success.

Structured adapters may be less important than parameter quantity.

Abstract

Full fine-tuning of large language models for alignment and task adaptation has become prohibitively expensive as models have grown in size. Parameter-Efficient Fine-Tuning (PEFT) methods aim at significantly reducing the computational and memory resources needed for fine-tuning these models by only training on a small number of parameters instead of all model parameters. Currently, the most popular PEFT method is the Low-Rank Adaptation (LoRA), which freezes the parameters of the model and introduces a small set of trainable parameters in the form of low-rank matrices. We propose simply reducing the number of trainable parameters by randomly selecting a small proportion of the model parameters to train on, while fixing all other parameters, without any additional prior assumptions such as low-rank structures. In this paper, we compare the efficiency and performance of our proposed approach to other PEFT methods as well as full parameter fine-tuning. We find our method to be competitive with LoRA when using a similar number of trainable parameters. Our findings suggest that what truly matters for a PEFT technique to perform well is not necessarily the specific adapter structure, but rather the number of trainable parameters being used.