A Closer Look at Conditional Prompt Tuning for Vision-Language Models

TL;DR

This paper identifies limitations in current conditional prompt tuning methods for vision-language models and introduces CaPT, a class-adaptive prompt tuning approach that improves generalization to new classes and enhances existing methods.

Contribution

The paper reveals the inefficacy of VII-conditioned prompts and proposes TCI-conditioned prompts with CaPT, boosting performance and generalization in vision-language models.

Findings

CaPT improves performance of unconditional PT baselines across 11 datasets.

CaPT can be integrated into existing PT schemes to mitigate BNT.

DeCaPT outperforms state-of-the-art conditional PT by 3.49% on average.

Abstract

Despite the great promise of Prompt Tuning (PT) in adapting large Vision-Language Pretrained Models (VLPMs) to downstream tasks, they often struggle to overcome the Base-New Tradeoff (BNT) dilemma: as VLPMs are better tuned to a base task, their ability to generalize to new tasks diminishes. Recent work on conditional PT addresses this problem by replacing static prompts with dynamic Visual Image Information (VII)-conditioned prompts, improving the model's generalization to new tasks to some extent. In this work, we first identify a critical issue with existing conditional PT methods: using VII as the "condition" of prompts yields suboptimal performance, and even random noise-conditioned prompts can outperform the VII-conditioned counterparts. On further analysis, we find that learning dynamic prompts conditioned on Textual Class Information (TCI) is the key to solving the BNT problem. Motivated by this, we then propose Class-adaptive Prompt Tuning (CaPT), which enables fast adaptation of tuned models to new classes by learning TCI-conditioned prompts from base classes. Remarkably, CaPT can be used as a plugin to mitigate the BNT problem for existing unconditional PT schemes. Extensive experiments on 11 datasets show that CaPT consistently improves the performance of five strong unconditional PT baselines with negligible additional computational cost. Additionally, by integrating CaPT with our recently proposed DePT framework, we devise a new conditional PT approach, termed DeCaPT, which outperforms the H ACC of the state-of-the-art conditional PT scheme by 3.49%, averaged over the 11 datasets. Code: https://github.com/Koorye/CaPT.