Dual Prompt Learning for Adapting Vision-Language Models to Downstream Image-Text Retrieval

TL;DR

This paper introduces DCAR, a dual prompt learning framework that enhances vision-language models for image-text retrieval by dynamically adjusting prompts to better discriminate fine-grained attributes and subcategories.

Contribution

The paper proposes a novel dual prompt learning approach with joint category-attribute reweighting to improve fine-grained image-text matching in downstream retrieval tasks.

Findings

DCAR achieves state-of-the-art results on the FDRD benchmark.

Dynamic prompt adjustment improves fine-grained attribute discrimination.

Joint optimization of attribute and category features enhances retrieval accuracy.

Abstract

Recently, prompt learning has demonstrated remarkable success in adapting pre-trained Vision-Language Models (VLMs) to various downstream tasks such as image classification. However, its application to the downstream Image-Text Retrieval (ITR) task is more challenging. We find that the challenge lies in discriminating both fine-grained attributes and similar subcategories of the downstream data. To address this challenge, we propose Dual prompt Learning with Joint Category-Attribute Reweighting (DCAR), a novel dual-prompt learning framework to achieve precise image-text matching. The framework dynamically adjusts prompt vectors from both semantic and visual dimensions to improve the performance of CLIP on the downstream ITR task. Based on the prompt paradigm, DCAR jointly optimizes attribute and class features to enhance fine-grained representation learning. Specifically, (1) at the attribute level, it dynamically updates the weights of attribute descriptions based on text-image mutual information correlation; (2) at the category level, it introduces negative samples from multiple perspectives with category-matching weighting to learn subcategory distinctions. To validate our method, we construct the Fine-class Described Retrieval Dataset (FDRD), which serves as a challenging benchmark for ITR in downstream data domains. It covers over 1,500 downstream fine categories and 230,000 image-caption pairs with detailed attribute annotations. Extensive experiments on FDRD demonstrate that DCAR achieves state-of-the-art performance over existing baselines.