ReID5o: Achieving Omni Multi-modal Person Re-identification in a Single Model

TL;DR

This paper introduces OM-ReID, a new multi-modal person re-identification challenge, along with ORBench dataset and ReID5o model, enabling effective retrieval across diverse modalities in a single framework.

Contribution

The paper presents the first high-quality multi-modal dataset ORBench and a novel ReID5o framework for omni multi-modal person re-identification in a unified model.

Findings

ReID5o achieves state-of-the-art performance on ORBench.

ORBench covers five diverse modalities, enhancing multi-modal ReID research.

ReID5o effectively fuses and aligns multiple modalities in a single model.

Abstract

In real-word scenarios, person re-identification (ReID) expects to identify a person-of-interest via the descriptive query, regardless of whether the query is a single modality or a combination of multiple modalities. However, existing methods and datasets remain constrained to limited modalities, failing to meet this requirement. Therefore, we investigate a new challenging problem called Omni Multi-modal Person Re-identification (OM-ReID), which aims to achieve effective retrieval with varying multi-modal queries. To address dataset scarcity, we construct ORBench, the first high-quality multi-modal dataset comprising 1,000 unique identities across five modalities: RGB, infrared, color pencil, sketch, and textual description. This dataset also has significant superiority in terms of diversity, such as the painting perspectives and textual information. It could serve as an ideal platform for follow-up investigations in OM-ReID. Moreover, we propose ReID5o, a novel multi-modal learning framework for person ReID. It enables synergistic fusion and cross-modal alignment of arbitrary modality combinations in a single model, with a unified encoding and multi-expert routing mechanism proposed. Extensive experiments verify the advancement and practicality of our ORBench. A wide range of possible models have been evaluated and compared on it, and our proposed ReID5o model gives the best performance. The dataset and code will be made publicly available at https://github.com/Zplusdragon/ReID5o_ORBench.