Gradient-based Parameter Selection for Efficient Fine-Tuning

TL;DR

This paper introduces Gradient-based Parameter Selection (GPS), a parameter-efficient fine-tuning method that tunes only a few parameters without extra computational costs, achieving comparable or better performance than full fine-tuning across various tasks.

Contribution

GPS is a novel, model-agnostic fine-tuning approach that selects a small subset of parameters to tune, eliminating additional costs and outperforming existing methods.

Findings

GPS improves accuracy by up to 9.61% on image classification tasks.

GPS achieves 17% improvement in mDice and 16.8% in mIoU on medical image segmentation.

GPS outperforms existing PEFT methods in various benchmarks.

Abstract

With the growing size of pre-trained models, full fine-tuning and storing all the parameters for various downstream tasks is costly and infeasible. In this paper, we propose a new parameter-efficient fine-tuning method, Gradient-based Parameter Selection (GPS), demonstrating that only tuning a few selected parameters from the pre-trained model while keeping the remainder of the model frozen can generate similar or better performance compared with the full model fine-tuning method. Different from the existing popular and state-of-the-art parameter-efficient fine-tuning approaches, our method does not introduce any additional parameters and computational costs during both the training and inference stages. Another advantage is the model-agnostic and non-destructive property, which eliminates the need for any other design specific to a particular model. Compared with the full fine-tuning, GPS achieves 3.33% (91.78% vs. 88.45%, FGVC) and 9.61% (73.1% vs. 65.57%, VTAB) improvement of the accuracy with tuning only 0.36% parameters of the pre-trained model on average over 24 image classification tasks; it also demonstrates a significant improvement of 17% and 16.8% in mDice and mIoU, respectively, on medical image segmentation task. Moreover, GPS achieves state-of-the-art performance compared with existing PEFT methods.