Selective Visual Prompting in Vision Mamba

TL;DR

This paper introduces Selective Visual Prompting (SVP), a novel fine-tuning method tailored for Vision Mamba models that enhances discriminative information propagation through token-wise prompts and dual-path structure.

Contribution

The paper proposes SVP, a new fine-tuning approach for Vim models that employs lightweight, token-wise prompts and a dual-path structure to improve information propagation and task performance.

Findings

SVP outperforms existing prompting methods on large-scale benchmarks.

SVP effectively activates input and forget gates in Vim, enhancing discriminative information flow.

Dual-path structure captures shared and specific layer information, boosting adaptability.

Abstract

Pre-trained Vision Mamba (Vim) models have demonstrated exceptional performance across various computer vision tasks in a computationally efficient manner, attributed to their unique design of selective state space models. To further extend their applicability to diverse downstream vision tasks, Vim models can be adapted using the efficient fine-tuning technique known as visual prompting. However, existing visual prompting methods are predominantly tailored for Vision Transformer (ViT)-based models that leverage global attention, neglecting the distinctive sequential token-wise compression and propagation characteristics of Vim. Specifically, existing prompt tokens prefixed to the sequence are insufficient to effectively activate the input and forget gates across the entire sequence, hindering the extraction and propagation of discriminative information. To address this limitation, we introduce a novel Selective Visual Prompting (SVP) method specifically for the efficient fine-tuning of Vim. To prevent the loss of discriminative information during state space propagation, SVP employs lightweight selective prompters for token-wise prompt generation, ensuring adaptive activation of the update and forget gates within Mamba blocks to promote discriminative information propagation. Moreover, considering that Vim propagates both shared cross-layer information and specific inner-layer information, we further refine SVP with a dual-path structure: Cross-Prompting and Inner-Prompting. Cross-Prompting utilizes shared parameters across layers, while Inner-Prompting employs distinct parameters, promoting the propagation of both shared and specific information, respectively. Extensive experimental results on various large-scale benchmarks demonstrate that our proposed SVP significantly outperforms state-of-the-art methods. Our code is available at https://github.com/zhoujiahuan1991/AAAI2025-SVP.