Convolutional Bypasses Are Better Vision Transformer Adapters

TL;DR

This paper introduces Convolutional Bypasses (Convpass), a lightweight adaptation module for Vision Transformers that leverages convolutional biases, improving transfer learning especially in low-data scenarios.

Contribution

It proposes Convpass, a novel convolution-based adaptation module for ViT, tailored for visual tasks and outperforming language-oriented modules in transfer learning.

Findings

Convpass uses less than 0.5% of parameters for adaptation.

Convpass outperforms existing PETL modules on VTAB-1K and few-shot datasets.

Convpass is particularly effective in low-data regimes.

Abstract

The pretrain-then-finetune paradigm has been widely adopted in computer vision. But as the size of Vision Transformer (ViT) grows exponentially, the full finetuning becomes prohibitive in view of the heavier storage overhead. Motivated by parameter-efficient transfer learning (PETL) on language transformers, recent studies attempt to insert lightweight adaptation modules (e.g., adapter layers or prompt tokens) to pretrained ViT and only finetune these modules while the pretrained weights are frozen. However, these modules were originally proposed to finetune language models and did not take into account the prior knowledge specifically for visual tasks. In this paper, we propose to construct Convolutional Bypasses (Convpass) in ViT as adaptation modules, introducing only a small amount (less than 0.5% of model parameters) of trainable parameters to adapt the large ViT. Different from other PETL methods, Convpass benefits from the hard-coded inductive bias of convolutional layers and thus is more suitable for visual tasks, especially in the low-data regime. Experimental results on VTAB-1K benchmark and few-shot learning datasets show that Convpass outperforms current language-oriented adaptation modules, demonstrating the necessity to tailor vision-oriented adaptation modules for adapting vision models.