Visual Instance-aware Prompt Tuning
Xi Xiao, Yunbei Zhang, Xingjian Li, Tianyang Wang, Xiao Wang, Yuxiang Wei, Jihun Hamm, Min Xu

TL;DR
This paper introduces ViaPT, a novel prompt tuning method for vision transformers that generates instance-aware prompts, improving performance by balancing dataset-level and instance-specific information, validated across 34 datasets.
Contribution
The paper proposes ViaPT, which generates instance-aware prompts using PCA, outperforming existing methods and providing a new paradigm for visual prompt optimization.
Findings
ViaPT outperforms state-of-the-art baselines on 34 datasets.
It effectively balances dataset-level and instance-specific prompts.
ViaPT reduces learnable parameters compared to VPT-Deep.
Abstract
Visual Prompt Tuning (VPT) has emerged as a parameter-efficient fine-tuning paradigm for vision transformers, with conventional approaches utilizing dataset-level prompts that remain the same across all input instances. We observe that this strategy results in sub-optimal performance due to high variance in downstream datasets. To address this challenge, we propose Visual Instance-aware Prompt Tuning (ViaPT), which generates instance-aware prompts based on each individual input and fuses them with dataset-level prompts, leveraging Principal Component Analysis (PCA) to retain important prompting information. Moreover, we reveal that VPT-Deep and VPT-Shallow represent two corner cases based on a conceptual understanding, in which they fail to effectively capture instance-specific information, while random dimension reduction on prompts only yields performance between the two extremes.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Advanced Neural Network Applications · Advanced Memory and Neural Computing
