Human-centered Interactive Learning via MLLMs for Text-to-Image Person Re-identification

TL;DR

This paper introduces an interactive learning framework using multimodal large language models to improve text-to-image person re-identification by refining queries and augmenting data, leading to significant performance gains.

Contribution

The paper proposes a novel human-centered interactive learning approach with a plug-and-play test-time module and a data augmentation strategy for enhanced TIReID performance.

Findings

Achieves state-of-the-art results on four TIReID benchmarks.

Effectively refines user queries through multimodal interactions.

Enhances training data quality with a new augmentation method.

Abstract

Despite remarkable advancements in text-to-image person re-identification (TIReID) facilitated by the breakthrough of cross-modal embedding models, existing methods often struggle to distinguish challenging candidate images due to intrinsic limitations, such as network architecture and data quality. To address these issues, we propose an Interactive Cross-modal Learning framework (ICL), which leverages human-centered interaction to enhance the discriminability of text queries through external multimodal knowledge. To achieve this, we propose a plug-and-play Test-time Humane-centered Interaction (THI) module, which performs visual question answering focused on human characteristics, facilitating multi-round interactions with a multimodal large language model (MLLM) to align query intent with latent target images. Specifically, THI refines user queries based on the MLLM responses to reduce the gap to the best-matching images, thereby boosting ranking accuracy. Additionally, to address the limitation of low-quality training texts, we introduce a novel Reorganization Data Augmentation (RDA) strategy based on information enrichment and diversity enhancement to enhance query discriminability by enriching, decomposing, and reorganizing person descriptions. Extensive experiments on four TIReID benchmarks, i.e., CUHK-PEDES, ICFG-PEDES, RSTPReid, and UFine6926, demonstrate that our method achieves remarkable performance with substantial improvement.