Exploring Visual Prompting: Robustness Inheritance and Beyond

TL;DR

This paper investigates how visual prompting inherits robustness from source models, identifies a robustness-generalization trade-off, and proposes a strategy called Prompt Boundary Loosening to improve robustness inheritance and generalization.

Contribution

It is the first to analyze robustness inheritance in visual prompting and introduces PBL, a strategy to mitigate robustness-generalization trade-offs in transfer learning.

Findings

Robust source models' robustness can be inherited by VP.

A trade-off exists between robustness and generalization in VP.

Prompt Boundary Loosening effectively enhances robustness inheritance and generalization.

Abstract

Visual Prompting (VP), an efficient method for transfer learning, has shown its potential in vision tasks. However, previous works focus exclusively on VP from standard source models, it is still unknown how it performs under the scenario of a robust source model: Can the robustness of the source model be successfully inherited? Does VP also encounter the same trade-off between robustness and generalization ability as the source model during this process? If such a trade-off exists, is there a strategy specifically tailored to VP to mitigate this limitation? In this paper, we thoroughly explore these three questions for the first time and provide affirmative answers to them. To mitigate the trade-off faced by VP, we propose a strategy called Prompt Boundary Loosening (PBL). As a lightweight, plug-and-play strategy naturally compatible with VP, PBL effectively ensures the successful inheritance of robustness when the source model is a robust model, while significantly enhancing VP's generalization ability across various downstream datasets. Extensive experiments across various datasets show that our findings are universal and demonstrate the significant benefits of the proposed strategy.