TAI++: Text as Image for Multi-Label Image Classification by Co-Learning Transferable Prompt

TL;DR

This paper introduces TAI++, a novel approach that transforms text into images to improve multi-label image classification by co-learning visual and textual prompts using pre-trained vision-language models.

Contribution

The paper proposes a pseudo-visual prompt module and a co-learning strategy with a dual-adapter to enhance visual knowledge transfer in multi-label classification.

Findings

Outperforms state-of-the-art methods on VOC2007, MS-COCO, and NUSWIDE datasets.

Effectively mines diverse visual knowledge via pseudo-visual prompts.

Enhances visual representation abilities through co-learning with text prompts.

Abstract

The recent introduction of prompt tuning based on pre-trained vision-language models has dramatically improved the performance of multi-label image classification. However, some existing strategies that have been explored still have drawbacks, i.e., either exploiting massive labeled visual data at a high cost or using text data only for text prompt tuning and thus failing to learn the diversity of visual knowledge. Hence, the application scenarios of these methods are limited. In this paper, we propose a pseudo-visual prompt~(PVP) module for implicit visual prompt tuning to address this problem. Specifically, we first learn the pseudo-visual prompt for each category, mining diverse visual knowledge by the well-aligned space of pre-trained vision-language models. Then, a co-learning strategy with a dual-adapter module is designed to transfer visual knowledge from pseudo-visual prompt to text prompt, enhancing their visual representation abilities. Experimental results on VOC2007, MS-COCO, and NUSWIDE datasets demonstrate that our method can surpass state-of-the-art~(SOTA) methods across various settings for multi-label image classification tasks. The code is available at https://github.com/njustkmg/PVP.