Tuning Multi-mode Token-level Prompt Alignment across Modalities

TL;DR

This paper introduces a multi-mode token-level prompt tuning framework for vision-language models, leveraging optimal transportation to improve semantic alignment and diversity across modalities, resulting in better generalization and few-shot learning.

Contribution

It proposes a novel multi-mode token-level prompt tuning method that captures diverse semantic representations and fine-grained alignment using optimal transportation, surpassing prior single-mode approaches.

Findings

Outperforms existing methods on image recognition benchmarks.

Enhances few-shot learning capabilities.

Learns prompt tokens that capture diverse visual concepts.

Abstract

Advancements in prompt tuning of vision-language models have underscored their potential in enhancing open-world visual concept comprehension. However, prior works only primarily focus on single-mode (only one prompt for each modality) and holistic level (image or sentence) semantic alignment, which fails to capture the sample diversity, leading to sub-optimal prompt discovery. To address the limitation, we propose a multi-mode token-level tuning framework that leverages the optimal transportation to learn and align a set of prompt tokens across modalities. Specifically, we rely on two essential factors: 1) multi-mode prompts discovery, which guarantees diverse semantic representations, and 2) token-level alignment, which helps explore fine-grained similarity. Consequently, the similarity can be calculated as a hierarchical transportation problem between the modality-specific sets. Extensive experiments on popular image recognition benchmarks show the superior generalization and few-shot abilities of our approach. The qualitative analysis demonstrates that the learned prompt tokens have the ability to capture diverse visual concepts.