All in One Framework for Multimodal Re-identification in the Wild

TL;DR

This paper introduces All-in-One (AIO), a pioneering unified framework for multimodal Re-identification that leverages a frozen pre-trained model to handle diverse data types without additional fine-tuning, excelling in zero-shot and domain generalization tasks.

Contribution

The paper presents the first all-in-one ReID framework that effectively integrates four modalities using a frozen large model and specialized cross-modality heads, without requiring fine-tuning.

Findings

AIO achieves superior performance across multiple modalities.

The framework excels in zero-shot and domain generalization scenarios.

It demonstrates robust retrieval accuracy on challenging multimodal datasets.

Abstract

In Re-identification (ReID), recent advancements yield noteworthy progress in both unimodal and cross-modal retrieval tasks. However, the challenge persists in developing a unified framework that could effectively handle varying multimodal data, including RGB, infrared, sketches, and textual information. Additionally, the emergence of large-scale models shows promising performance in various vision tasks but the foundation model in ReID is still blank. In response to these challenges, a novel multimodal learning paradigm for ReID is introduced, referred to as All-in-One (AIO), which harnesses a frozen pre-trained big model as an encoder, enabling effective multimodal retrieval without additional fine-tuning. The diverse multimodal data in AIO are seamlessly tokenized into a unified space, allowing the modality-shared frozen encoder to extract identity-consistent features comprehensively across all modalities. Furthermore, a meticulously crafted ensemble of cross-modality heads is designed to guide the learning trajectory. AIO is the \textbf{first} framework to perform all-in-one ReID, encompassing four commonly used modalities. Experiments on cross-modal and multimodal ReID reveal that AIO not only adeptly handles various modal data but also excels in challenging contexts, showcasing exceptional performance in zero-shot and domain generalization scenarios.