SCT: A Simple Baseline for Parameter-Efficient Fine-Tuning via Salient Channels

TL;DR

SCT is a simple, parameter-efficient fine-tuning method for vision transformers that selectively tunes salient channels, outperforming full fine-tuning on most tasks while using significantly fewer parameters.

Contribution

We introduce Salient Channel Tuning (SCT), a novel approach that leverages task-specific salient channels to enable effective fine-tuning with only 1/8 of the parameters.

Findings

Outperforms full fine-tuning on 18 of 19 tasks

Uses only 0.11M parameters, 780x fewer than full fine-tuning

Effective in domain generalization and few-shot classification

Abstract

Pre-trained vision transformers have strong representation benefits to various downstream tasks. Recently, many parameter-efficient fine-tuning (PEFT) methods have been proposed, and their experiments demonstrate that tuning only 1\% extra parameters could surpass full fine-tuning in low-data resource scenarios. However, these methods overlook the task-specific information when fine-tuning diverse downstream tasks. In this paper, we propose a simple yet effective method called "Salient Channel Tuning" (SCT) to leverage the task-specific information by forwarding the model with the task images to select partial channels in a feature map that enables us to tune only 1/8 channels leading to significantly lower parameter costs. Experiments on 19 visual transfer learning downstream tasks demonstrate that our SCT outperforms full fine-tuning on 18 out of 19 tasks by adding only 0.11M parameters of the ViT-B, which is 780$\times$ fewer than its full fine-tuning counterpart. Furthermore, experiments on domain generalization and few-shot classification further demonstrate the effectiveness and generic of our approach. The code is available at https://github.com/showlab/SCT.