Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?

TL;DR

This paper analyzes when and why Visual Prompt Tuning (VPT) outperforms full fine-tuning in vision models, revealing it is preferable under certain task and data conditions and due to its feature-preserving mechanism.

Contribution

It offers a comprehensive analysis across multiple datasets to clarify the conditions favoring VPT and investigates the underlying reasons for its effectiveness.

Findings

VPT is preferable when task objectives differ significantly or data distributions are similar.

VPT's success is linked to its ability to preserve original features and add parameters.

Overfitting and optimization are not the sole reasons for VPT's effectiveness.

Abstract

As the scale of vision models continues to grow, the emergence of Visual Prompt Tuning (VPT) as a parameter-efficient transfer learning technique has gained attention due to its superior performance compared to traditional full-finetuning. However, the conditions favoring VPT (the ``when") and the underlying rationale (the ``why") remain unclear. In this paper, we conduct a comprehensive analysis across 19 distinct datasets and tasks. To understand the ``when" aspect, we identify the scenarios where VPT proves favorable by two dimensions: task objectives and data distributions. We find that VPT is preferrable when there is 1) a substantial disparity between the original and the downstream task objectives (e.g., transitioning from classification to counting), or 2) a similarity in data distributions between the two tasks (e.g., both involve natural images). In exploring the ``why" dimension, our results indicate VPT's success cannot be attributed solely to overfitting and optimization considerations. The unique way VPT preserves original features and adds parameters appears to be a pivotal factor. Our study provides insights into VPT's mechanisms, and offers guidance for its optimal utilization.