When Large Vision-Language Models Meet Person Re-Identification

TL;DR

This paper introduces LVLM-ReID, a framework that leverages large vision-language models to improve person re-identification by generating and refining semantic tokens representing identity features.

Contribution

It proposes a novel method to utilize LVLMs for ReID by generating semantic tokens guided by instructions and refined through a Semantic-Guided Interaction module.

Findings

Achieves competitive results on multiple ReID benchmarks.

Operates without additional image-text annotations.

Effectively captures rich semantic cues for identity recognition.

Abstract

Large Vision-Language Models (LVLMs) that incorporate visual models and large language models have achieved impressive results across cross-modal understanding and reasoning tasks. In recent years, person re-identification (ReID) has also started to explore cross-modal semantics to improve the accuracy of identity recognition. However, effectively utilizing LVLMs for ReID remains an open challenge. While LVLMs operate under a generative paradigm by predicting the next output word, ReID requires the extraction of discriminative identity features to match pedestrians across cameras. In this paper, we propose LVLM-ReID, a novel framework that harnesses the strengths of LVLMs to promote ReID. Specifically, we employ instructions to guide the LVLM in generating one semantic token that encapsulates key appearance semantics from the person image. This token is further refined through our Semantic-Guided Interaction (SGI) module, establishing a reciprocal interaction between the semantic token and visual tokens. Ultimately, the reinforced semantic token serves as the representation of pedestrian identity. Our framework integrates the semantic understanding and generation capabilities of LVLM into end-to-end ReID training, allowing LVLM to capture rich semantic cues during both training and inference. LVLM-ReID achieves competitive results on multiple benchmarks without additional image-text annotations, demonstrating the potential of LVLM-generated semantics to advance person ReID.