Generalizable Prompt Tuning for Vision-Language Models

TL;DR

This paper proposes a novel prompt tuning method for vision-language models that enhances both task-specific performance and generalization to unseen classes by leveraging dual views of prompts and visual augmentation.

Contribution

It introduces a mutual information maximization approach between soft and hand-crafted prompts and incorporates class-wise visual augmentation for improved generalization.

Findings

Achieves competitive task-specific performance on benchmarks.

Demonstrates improved generalization to unseen classes.

Shows robustness through visual modality augmentation.

Abstract

Prompt tuning for vision-language models such as CLIP involves optimizing the text prompts used to generate image-text pairs for specific downstream tasks. While hand-crafted or template-based prompts are generally applicable to a wider range of unseen classes, they tend to perform poorly in downstream tasks (i.e., seen classes). Learnable soft prompts, on the other hand, often perform well in downstream tasks but lack generalizability. Additionally, prior research has predominantly concentrated on the textual modality, with very few studies attempting to explore the prompt's generalization potential from the visual modality. Keeping these limitations in mind, we investigate how to prompt tuning to obtain both a competitive downstream performance and generalization. The study shows that by treating soft and hand-crafted prompts as dual views of the textual modality, and maximizing their mutual information, we can better ensemble task-specific and general semantic information. Moreover, to generate more expressive prompts, the study introduces a class-wise augmentation from the visual modality, resulting in significant robustness to a wider range of unseen classes. Extensive evaluations on several benchmarks report that the proposed approach achieves competitive results in terms of both task-specific performance and general abilities.