Visual Instance-aware Prompt Tuning

TL;DR

This paper introduces ViaPT, a novel prompt tuning method for vision transformers that generates instance-aware prompts, improving performance by balancing dataset-level and instance-specific information, validated across 34 datasets.

Contribution

The paper proposes ViaPT, which generates instance-aware prompts using PCA, outperforming existing methods and providing a new paradigm for visual prompt optimization.

Findings

ViaPT outperforms state-of-the-art baselines on 34 datasets.

It effectively balances dataset-level and instance-specific prompts.

ViaPT reduces learnable parameters compared to VPT-Deep.

Abstract

Visual Prompt Tuning (VPT) has emerged as a parameter-efficient fine-tuning paradigm for vision transformers, with conventional approaches utilizing dataset-level prompts that remain the same across all input instances. We observe that this strategy results in sub-optimal performance due to high variance in downstream datasets. To address this challenge, we propose Visual Instance-aware Prompt Tuning (ViaPT), which generates instance-aware prompts based on each individual input and fuses them with dataset-level prompts, leveraging Principal Component Analysis (PCA) to retain important prompting information. Moreover, we reveal that VPT-Deep and VPT-Shallow represent two corner cases based on a conceptual understanding, in which they fail to effectively capture instance-specific information, while random dimension reduction on prompts only yields performance between the two extremes. Instead, ViaPT overcomes these limitations by balancing dataset-level and instance-level knowledge, while reducing the amount of learnable parameters compared to VPT-Deep. Extensive experiments across 34 diverse datasets demonstrate that our method consistently outperforms state-of-the-art baselines, establishing a new paradigm for analyzing and optimizing visual prompts for vision transformers.