Enhancing Visible-Infrared Person Re-identification with Modality- and Instance-aware Visual Prompt Learning

TL;DR

This paper introduces a transformer-based Modality-aware and Instance-aware Visual Prompt (MIP) network for visible-infrared person re-identification, effectively utilizing both invariant and modality-specific features to improve matching accuracy across modalities.

Contribution

The paper proposes a novel MIP model with modality-specific and instance-specific prompts to reduce modality gap and enhance discriminative features in VI ReID tasks.

Findings

MIP outperforms most state-of-the-art methods on SYSU-MM01 and RegDB datasets.

Designed prompts effectively reduce modality interference and improve identification accuracy.

Extensive experiments validate the effectiveness of modality- and instance-aware prompts.

Abstract

The Visible-Infrared Person Re-identification (VI ReID) aims to match visible and infrared images of the same pedestrians across non-overlapped camera views. These two input modalities contain both invariant information, such as shape, and modality-specific details, such as color. An ideal model should utilize valuable information from both modalities during training for enhanced representational capability. However, the gap caused by modality-specific information poses substantial challenges for the VI ReID model to handle distinct modality inputs simultaneously. To address this, we introduce the Modality-aware and Instance-aware Visual Prompts (MIP) network in our work, designed to effectively utilize both invariant and specific information for identification. Specifically, our MIP model is built on the transformer architecture. In this model, we have designed a series of modality-specific prompts, which could enable our model to adapt to and make use of the specific information inherent in different modality inputs, thereby reducing the interference caused by the modality gap and achieving better identification. Besides, we also employ each pedestrian feature to construct a group of instance-specific prompts. These customized prompts are responsible for guiding our model to adapt to each pedestrian instance dynamically, thereby capturing identity-level discriminative clues for identification. Through extensive experiments on SYSU-MM01 and RegDB datasets, the effectiveness of both our designed modules is evaluated. Additionally, our proposed MIP performs better than most state-of-the-art methods.