Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Model

TL;DR

This paper introduces Dude, a dual distribution-aware prompt learning framework for large vision-language models that leverages domain-shared and class-specific contexts, enhanced by optimal transport theory, to improve fine-grained classification performance.

Contribution

The paper proposes a novel dual prompt framework combined with Unbalanced Optimal Transport to better align visual tokens and prompts, addressing limitations of existing prompt methods in fine-grained tasks.

Findings

Outperforms state-of-the-art baselines in few-shot classification.

Effectively handles noisy and irrelevant elements through UOT-based partial matching.

Enhances feature representation with dual context prompts and optimal transport.

Abstract

Prompt learning methods are gaining increasing attention due to their ability to customize large vision-language models to new domains using pre-trained contextual knowledge and minimal training data. However, existing works typically rely on optimizing unified prompt inputs, often struggling with fine-grained classification tasks due to insufficient discriminative attributes. To tackle this, we consider a new framework based on a dual context of both domain-shared and class-specific contexts, where the latter is generated by Large Language Models (LLMs) such as GPTs. Such dual prompt methods enhance the model's feature representation by joining implicit and explicit factors encoded in LLM knowledge. Moreover, we formulate the Unbalanced Optimal Transport (UOT) theory to quantify the relationships between constructed prompts and visual tokens. Through partial matching, UOT can properly align discrete sets of visual tokens and prompt embeddings under different mass distributions, which is particularly valuable for handling irrelevant or noisy elements, ensuring that the preservation of mass does not restrict transport solutions. Furthermore, UOT's characteristics integrate seamlessly with image augmentation, expanding the training sample pool while maintaining a reasonable distance between perturbed images and prompt inputs. Extensive experiments across few-shot classification and adapter settings substantiate the superiority of our model over current state-of-the-art baselines.