PVLR: Prompt-driven Visual-Linguistic Representation Learning for Multi-Label Image Recognition

TL;DR

This paper introduces PVLR, a novel framework for multi-label image recognition that leverages dual-prompting and bidirectional fusion of visual and linguistic features to improve label prediction accuracy.

Contribution

The paper proposes a dual-prompting strategy and a bidirectional fusion module to better utilize language knowledge and enhance multi-label recognition performance.

Findings

PVLR outperforms existing methods on MS-COCO, Pascal VOC 2007, and NUS-WIDE datasets.

The dual-prompting strategy effectively captures label semantics and relationships.

Bidirectional fusion improves the interaction between visual and linguistic features.

Abstract

Multi-label image recognition is a fundamental task in computer vision. Recently, vision-language models have made notable advancements in this area. However, previous methods often failed to effectively leverage the rich knowledge within language models and instead incorporated label semantics into visual features in a unidirectional manner. In this paper, we propose a Prompt-driven Visual-Linguistic Representation Learning (PVLR) framework to better leverage the capabilities of the linguistic modality. In PVLR, we first introduce a dual-prompting strategy comprising Knowledge-Aware Prompting (KAP) and Context-Aware Prompting (CAP). KAP utilizes fixed prompts to capture the intrinsic semantic knowledge and relationships across all labels, while CAP employs learnable prompts to capture context-aware label semantics and relationships. Later, we propose an Interaction and Fusion Module (IFM) to interact and fuse the representations obtained from KAP and CAP. In contrast to the unidirectional fusion in previous works, we introduce a Dual-Modal Attention (DMA) that enables bidirectional interaction between textual and visual features, yielding context-aware label representations and semantic-related visual representations, which are subsequently used to calculate similarities and generate final predictions for all labels. Extensive experiments on three popular datasets including MS-COCO, Pascal VOC 2007, and NUS-WIDE demonstrate the superiority of PVLR.