Gradient-based Fine-Tuning through Pre-trained Model Regularization

TL;DR

This paper introduces GRFT, a gradient-based regularized fine-tuning method for large pre-trained models that updates minimal parameters, reducing resource needs while achieving state-of-the-art results.

Contribution

The paper presents a novel efficient fine-tuning approach that updates only specific rows or columns of weight matrices with regularization, outperforming existing methods.

Findings

GRFT updates only 1.22% and 0.30% of parameters on FGVC and VTAB datasets.

It surpasses GPS, Adapter Tuning, and LoRA in performance.

The method reduces storage and computational requirements significantly.

Abstract

Large pre-trained models have demonstrated extensive applications across various fields. However, fine-tuning these models for specific downstream tasks demands significant computational resources and storage. One fine-tuning method, gradient-based parameter selection (GPS), focuses on fine-tuning only the parameters with high gradients in each neuron, thereby reducing the number of training parameters. Nevertheless, this approach increases computational resource requirements and storage demands. In this paper, we propose an efficient gradient-based and regularized fine-tuning method (GRFT) that updates the rows or columns of the weight matrix. We theoretically demonstrate that the rows or columns with the highest sum of squared gradients are optimal for updating. This strategy effectively reduces storage overhead and improves the efficiency of parameter selection. Additionally, we incorporate regularization to enhance knowledge transfer from the pre-trained model. GRFT achieves state-of-the-art performance, surpassing existing methods such as GPS, Adapter Tuning, and LoRA. Notably, GRFT requires updating only 1.22% and 0.30% of the total parameters on FGVC and VTAB datasets, respectively, demonstrating its high efficiency and effectiveness. The source code will be released soon.