BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning

TL;DR

BlackVIP introduces a novel black-box visual prompting method that enables robust transfer learning from large pre-trained models without requiring access to model parameters, making it practical for real-world applications with limited memory.

Contribution

It proposes a parameter-free, black-box visual prompting approach with a gradient estimation technique, expanding transfer learning capabilities to proprietary or API-based models.

Findings

Effective on 16 datasets, demonstrating robustness across diverse domains.

Requires minimal memory, suitable for real-world deployment.

Outperforms existing methods in black-box transfer learning scenarios.

Abstract

With the surge of large-scale pre-trained models (PTMs), fine-tuning these models to numerous downstream tasks becomes a crucial problem. Consequently, parameter efficient transfer learning (PETL) of large models has grasped huge attention. While recent PETL methods showcase impressive performance, they rely on optimistic assumptions: 1) the entire parameter set of a PTM is available, and 2) a sufficiently large memory capacity for the fine-tuning is equipped. However, in most real-world applications, PTMs are served as a black-box API or proprietary software without explicit parameter accessibility. Besides, it is hard to meet a large memory requirement for modern PTMs. In this work, we propose black-box visual prompting (BlackVIP), which efficiently adapts the PTMs without knowledge about model architectures and parameters. BlackVIP has two components; 1) Coordinator and 2) simultaneous perturbation stochastic approximation with gradient correction (SPSA-GC). The Coordinator designs input-dependent image-shaped visual prompts, which improves few-shot adaptation and robustness on distribution/location shift. SPSA-GC efficiently estimates the gradient of a target model to update Coordinator. Extensive experiments on 16 datasets demonstrate that BlackVIP enables robust adaptation to diverse domains without accessing PTMs' parameters, with minimal memory requirements. Code: \url{https://github.com/changdaeoh/BlackVIP}