Visual Prompt Tuning

TL;DR

Visual Prompt Tuning (VPT) offers a parameter-efficient method for adapting large vision models by tuning only a small input space component, often outperforming traditional full fine-tuning across various tasks.

Contribution

VPT introduces a novel approach that tunes less than 1% of parameters in vision transformers, maintaining a frozen backbone while achieving superior or comparable performance.

Findings

VPT outperforms other tuning methods on multiple vision tasks.

VPT can surpass full fine-tuning in many scenarios.

VPT reduces storage costs per task.

Abstract

The current modus operandi in adapting pre-trained models involves updating all the backbone parameters, ie, full fine-tuning. This paper introduces Visual Prompt Tuning (VPT) as an efficient and effective alternative to full fine-tuning for large-scale Transformer models in vision. Taking inspiration from recent advances in efficiently tuning large language models, VPT introduces only a small amount (less than 1% of model parameters) of trainable parameters in the input space while keeping the model backbone frozen. Via extensive experiments on a wide variety of downstream recognition tasks, we show that VPT achieves significant performance gains compared to other parameter efficient tuning protocols. Most importantly, VPT even outperforms full fine-tuning in many cases across model capacities and training data scales, while reducing per-task storage cost.