Image-Text Knowledge Modeling for Unsupervised Multi-Scenario Person Re-Identification

TL;DR

This paper introduces a novel unsupervised multi-scenario person re-identification framework leveraging vision-language models, significantly improving cross-scenario matching accuracy by integrating image-text knowledge modeling.

Contribution

It proposes a three-stage framework, ITKM, that adaptively leverages CLIP's vision-language capabilities for multi-scenario person ReID, including scenario embedding, text embedding optimization, and heterogeneous matching modules.

Findings

Outperforms existing scenario-specific methods.

Enhances overall ReID performance across diverse scenarios.

Demonstrates strong generalizability and effectiveness.

Abstract

We propose unsupervised multi-scenario (UMS) person re-identification (ReID) as a new task that expands ReID across diverse scenarios (cross-resolution, clothing change, etc.) within a single coherent framework. To tackle UMS-ReID, we introduce image-text knowledge modeling (ITKM) -- a three-stage framework that effectively exploits the representational power of vision-language models. We start with a pre-trained CLIP model with an image encoder and a text encoder. In Stage I, we introduce a scenario embedding in the image encoder and fine-tune the encoder to adaptively leverage knowledge from multiple scenarios. In Stage II, we optimize a set of learned text embeddings to associate with pseudo-labels from Stage I and introduce a multi-scenario separation loss to increase the divergence between inter-scenario text representations. In Stage III, we first introduce cluster-level and instance-level heterogeneous matching modules to obtain reliable heterogeneous positive pairs (e.g., a visible image and an infrared image of the same person) within each scenario. Next, we propose a dynamic text representation update strategy to maintain consistency between text and image supervision signals. Experimental results across multiple scenarios demonstrate the superiority and generalizability of ITKM; it not only outperforms existing scenario-specific methods but also enhances overall performance by integrating knowledge from multiple scenarios.