Visual prompting reimagined: The power of the Activation Prompts
Yihua Zhang, Hongkang Li, Yuguang Yao, Aochuan Chen, Shuai Zhang, Pin-Yu Chen, Meng Wang, Sijia Liu

TL;DR
This paper introduces activation prompts (AP), a generalized form of visual prompting that applies perturbations to intermediate activation maps, significantly improving performance and efficiency over traditional input-level VP and fine-tuning methods.
Contribution
The paper proposes activation prompts (AP), extending visual prompting to internal model layers, and demonstrates its advantages through theoretical analysis and extensive experiments.
Findings
AP outperforms VP in accuracy and efficiency across 29 datasets.
AP reveals model-dependent layer preferences for prompting.
Theoretical analysis explains the layer-specific effectiveness of AP.
Abstract
Visual prompting (VP) has emerged as a popular method to repurpose pretrained vision models for adaptation to downstream tasks. Unlike conventional model fine-tuning techniques, VP introduces a universal perturbation directly into the input data to facilitate task-specific fine-tuning rather than modifying model parameters. However, there exists a noticeable performance gap between VP and conventional fine-tuning methods, highlighting an unexplored realm in theory and practice to understand and advance the input-level VP to reduce its current performance gap. Towards this end, we introduce a generalized concept, termed activation prompt (AP), which extends the scope of the input-level VP by enabling universal perturbations to be applied to activation maps within the intermediate layers of the model. By using AP to revisit the problem of VP and employing it as an analytical tool, we…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
