Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning

TL;DR

This paper introduces SSF, a simple yet effective parameter-efficient fine-tuning method that scales and shifts features, achieving comparable or better performance than full fine-tuning with fewer parameters and no extra inference cost.

Contribution

The paper proposes SSF, a novel fine-tuning approach that only adjusts feature scaling and shifting, outperforming existing methods while adding minimal parameters and enabling re-parameterization for efficient inference.

Findings

SSF outperforms full fine-tuning on FGVC and VTAB-1k datasets.

SSF requires tuning only about 0.3M parameters.

Extensive experiments validate SSF's effectiveness across various models and datasets.

Abstract

Existing fine-tuning methods either tune all parameters of the pre-trained model (full fine-tuning), which is not efficient, or only tune the last linear layer (linear probing), which suffers a significant accuracy drop compared to the full fine-tuning. In this paper, we propose a new parameter-efficient fine-tuning method termed as SSF, representing that researchers only need to Scale and Shift the deep Features extracted by a pre-trained model to catch up with the performance of full fine-tuning. In this way, SSF also surprisingly outperforms other parameter-efficient fine-tuning approaches even with a smaller number of tunable parameters. Furthermore, different from some existing parameter-efficient fine-tuning methods (e.g., Adapter or VPT) that introduce the extra parameters and computational cost in the training and inference stages, SSF only adds learnable parameters during the training stage, and these additional parameters can be merged into the original pre-trained model weights via re-parameterization in the inference phase. With the proposed SSF, our model obtains 2.46% (90.72% vs. 88.54%) and 11.48% (73.10% vs. 65.57%) performance improvement on FGVC and VTAB-1k in terms of Top-1 accuracy compared to the full fine-tuning but only fine-tuning about 0.3M parameters. We also conduct amounts of experiments in various model families (CNNs, Transformers, and MLPs) and datasets. Results on 26 image classification datasets in total and 3 robustness & out-of-distribution datasets show the effectiveness of SSF. Code is available at https://github.com/dongzelian/SSF.