Decouple before Align: Visual Disentanglement Enhances Prompt Tuning

TL;DR

This paper introduces DAPT, a prompt tuning framework that explicitly decouples visual features into foreground and background to improve alignment with textual data, leading to better performance in vision-language tasks.

Contribution

The paper proposes a novel decouple-before-align approach for prompt tuning, explicitly separating visual features into foreground and background to enhance modality alignment.

Findings

DAPT improves few-shot learning performance.

DAPT enhances base-to-novel generalization.

DAPT achieves superior results on benchmarks.

Abstract

Prompt tuning (PT), as an emerging resource-efficient fine-tuning paradigm, has showcased remarkable effectiveness in improving the task-specific transferability of vision-language models. This paper delves into a previously overlooked information asymmetry issue in PT, where the visual modality mostly conveys more context than the object-oriented textual modality. Correspondingly, coarsely aligning these two modalities could result in the biased attention, driving the model to merely focus on the context area. To address this, we propose DAPT, an effective PT framework based on an intuitive decouple-before-align concept. First, we propose to explicitly decouple the visual modality into the foreground and background representation via exploiting coarse-and-fine visual segmenting cues, and then both of these decoupled patterns are aligned with the original foreground texts and the hand-crafted background classes, thereby symmetrically strengthening the modal alignment. To further enhance the visual concentration, we propose a visual pull-push regularization tailored for the foreground-background patterns, directing the original visual representation towards unbiased attention on the region-of-interest object. We demonstrate the power of architecture-free DAPT through few-shot learning, base-to-novel generalization, and data-efficient learning, all of which yield superior performance across prevailing benchmarks. Our code will be released at https://github.com/Ferenas/DAPT.