AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition

TL;DR

AdaptFormer is a lightweight, plug-and-play method that efficiently adapts pre-trained Vision Transformers to various image and video recognition tasks, significantly improving transferability without full fine-tuning.

Contribution

It introduces lightweight modules that enhance ViT transferability with minimal additional parameters, outperforming fully fine-tuned models across multiple visual recognition benchmarks.

Findings

Achieves about 10% and 19% relative improvement on specific video datasets.

Adds less than 2% extra parameters to ViTs.

Largely improves ViT performance in target domains.

Abstract

Pretraining Vision Transformers (ViTs) has achieved great success in visual recognition. A following scenario is to adapt a ViT to various image and video recognition tasks. The adaptation is challenging because of heavy computation and memory storage. Each model needs an independent and complete finetuning process to adapt to different tasks, which limits its transferability to different visual domains. To address this challenge, we propose an effective adaptation approach for Transformer, namely AdaptFormer, which can adapt the pre-trained ViTs into many different image and video tasks efficiently. It possesses several benefits more appealing than prior arts. Firstly, AdaptFormer introduces lightweight modules that only add less than 2% extra parameters to a ViT, while it is able to increase the ViT's transferability without updating its original pre-trained parameters, significantly outperforming the existing 100\% fully fine-tuned models on action recognition benchmarks. Secondly, it can be plug-and-play in different Transformers and scalable to many visual tasks. Thirdly, extensive experiments on five image and video datasets show that AdaptFormer largely improves ViTs in the target domains. For example, when updating just 1.5% extra parameters, it achieves about 10% and 19% relative improvement compared to the fully fine-tuned models on Something-Something~v2 and HMDB51, respectively. Code is available at https://github.com/ShoufaChen/AdaptFormer.