Multi-Prompts Learning with Cross-Modal Alignment for Attribute-based Person Re-Identification

TL;DR

This paper introduces MP-ReID, a novel framework that leverages generated fine-grained attribute prompts and language models to improve person re-identification accuracy by effectively utilizing rich contextual attribute information.

Contribution

It proposes a new prompt learning framework that uses explicit and implicit attribute prompts with cross-modal alignment for enhanced person ReID performance.

Findings

Significant improvement on Market1501 and DukeMTMC-reID datasets.

Effective fusion of explicit and implicit prompts reduces cross-modal gaps.

Demonstrates the potential of large language models in attribute-based person ReID.

Abstract

The fine-grained attribute descriptions can significantly supplement the valuable semantic information for person image, which is vital to the success of person re-identification (ReID) task. However, current ReID algorithms typically failed to effectively leverage the rich contextual information available, primarily due to their reliance on simplistic and coarse utilization of image attributes. Recent advances in artificial intelligence generated content have made it possible to automatically generate plentiful fine-grained attribute descriptions and make full use of them. Thereby, this paper explores the potential of using the generated multiple person attributes as prompts in ReID tasks with off-the-shelf (large) models for more accurate retrieval results. To this end, we present a new framework called Multi-Prompts ReID (MP-ReID), based on prompt learning and language models, to fully dip fine attributes to assist ReID task. Specifically, MP-ReID first learns to hallucinate diverse, informative, and promptable sentences for describing the query images. This procedure includes (i) explicit prompts of which attributes a person has and furthermore (ii) implicit learnable prompts for adjusting/conditioning the criteria used towards this person identity matching. Explicit prompts are obtained by ensembling generation models, such as ChatGPT and VQA models. Moreover, an alignment module is designed to fuse multi-prompts (i.e., explicit and implicit ones) progressively and mitigate the cross-modal gap. Extensive experiments on the existing attribute-involved ReID datasets, namely, Market1501 and DukeMTMC-reID, demonstrate the effectiveness and rationality of the proposed MP-ReID solution.