Context-Aware Prompt Tuning for Vision-Language Model with Dual-Alignment

TL;DR

This paper proposes Dual-Aligned Prompt Tuning (DuAl-PT), a novel method that combines explicit context descriptions from large language models with implicit prompt learning to enhance vision-language model adaptation in few-shot tasks.

Contribution

The paper introduces DuAl-PT, which aligns prompts with both LLM-generated context and image features, improving few-shot learning and generalization in vision-language models.

Findings

Achieves superior performance on 11 datasets in few-shot recognition.

Enhances base-to-new generalization capabilities.

Serves as a strong baseline for prompt tuning methods.

Abstract

Large-scale vision-language models (VLMs), e.g., CLIP, learn broad visual concepts from tedious training data, showing superb generalization ability. Amount of prompt learning methods have been proposed to efficiently adapt the VLMs to downstream tasks with only a few training samples. We introduce a novel method to improve the prompt learning of vision-language models by incorporating pre-trained large language models (LLMs), called Dual-Aligned Prompt Tuning (DuAl-PT). Learnable prompts, like CoOp, implicitly model the context through end-to-end training, which are difficult to control and interpret. While explicit context descriptions generated by LLMs, like GPT-3, can be directly used for zero-shot classification, such prompts are overly relying on LLMs and still underexplored in few-shot domains. With DuAl-PT, we propose to learn more context-aware prompts, benefiting from both explicit and implicit context modeling. To achieve this, we introduce a pre-trained LLM to generate context descriptions, and we encourage the prompts to learn from the LLM's knowledge by alignment, as well as the alignment between prompts and local image features. Empirically, DuAl-PT achieves superior performance on 11 downstream datasets on few-shot recognition and base-to-new generalization. Hopefully, DuAl-PT can serve as a strong baseline. Code will be available.