LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation

TL;DR

LoR-VP introduces a low-rank visual prompting method that enhances the efficiency and performance of adapting pre-trained vision models by enabling shared and patch-specific information, significantly reducing parameters and training time.

Contribution

The paper proposes a novel low-rank visual prompting technique that addresses limitations of existing methods, improving both efficiency and effectiveness in model adaptation.

Findings

Up to 6 times faster training

Utilizes 18 times fewer prompt parameters

Achieves 3.1% performance improvement

Abstract

Visual prompting has gained popularity as a method for adapting pre-trained models to specific tasks, particularly in the realm of parameter-efficient tuning. However, existing visual prompting techniques often pad the prompt parameters around the image, limiting the interaction between the visual prompts and the original image to a small set of patches while neglecting the inductive bias present in shared information across different patches. In this study, we conduct a thorough preliminary investigation to identify and address these limitations. We propose a novel visual prompt design, introducing Low-Rank matrix multiplication for Visual Prompting (LoR-VP), which enables shared and patch-specific information across rows and columns of image pixels. Extensive experiments across seven network architectures and four datasets demonstrate significant improvements in both performance and efficiency compared to state-of-the-art visual prompting methods, achieving up to 6 times faster training times, utilizing 18 times fewer visual prompt parameters, and delivering a 3.1% improvement in performance. The code is available as https://github.com/jincan333/LoR-VP.