Dynamic Visual Prompt Tuning for Parameter Efficient Transfer Learning

TL;DR

This paper introduces DVPT, a dynamic visual prompt tuning method that generates instance-specific prompts for each image, improving transfer learning efficiency and performance on visual tasks.

Contribution

The paper proposes a novel dynamic prompt generation framework using a Meta-Net, enabling instance-wise adaptation in parameter-efficient transfer learning.

Findings

DVPT outperforms existing PETL methods on various tasks.

DVPT surpasses full fine-tuning on 17 out of 19 tasks.

The approach maintains high parameter efficiency.

Abstract

Parameter efficient transfer learning (PETL) is an emerging research spot that aims to adapt large-scale pre-trained models to downstream tasks. Recent advances have achieved great success in saving storage and computation costs. However, these methods do not take into account instance-specific visual clues for visual tasks. In this paper, we propose a Dynamic Visual Prompt Tuning framework (DVPT), which can generate a dynamic instance-wise token for each image. In this way, it can capture the unique visual feature of each image, which can be more suitable for downstream visual tasks. We designed a Meta-Net module that can generate learnable prompts based on each image, thereby capturing dynamic instance-wise visual features. Extensive experiments on a wide range of downstream recognition tasks show that DVPT achieves superior performance than other PETL methods. More importantly, DVPT even outperforms full fine-tuning on 17 out of 19 downstream tasks while maintaining high parameter efficiency. Our code will be released soon.