Prototypical Contrastive Learning-based CLIP Fine-tuning for Object Re-identification

TL;DR

This paper introduces a simple prototypical contrastive learning method to fine-tune CLIP for object re-identification, outperforming prompt learning approaches in supervised and unsupervised settings.

Contribution

It proposes a novel PCL-based fine-tuning approach for CLIP that removes prompt learning and improves Re-ID performance in various supervision scenarios.

Findings

Outperforms CLIP-ReID in supervised Re-ID tasks

Achieves state-of-the-art results in unsupervised Re-ID

Eliminates the need for prompt learning in CLIP fine-tuning

Abstract

This work aims to adapt large-scale pre-trained vision-language models, such as contrastive language-image pretraining (CLIP), to enhance the performance of object reidentification (Re-ID) across various supervision settings. Although prompt learning has enabled a recent work named CLIP-ReID to achieve promising performance, the underlying mechanisms and the necessity of prompt learning remain unclear due to the absence of semantic labels in ReID tasks. In this work, we first analyze the role prompt learning in CLIP-ReID and identify its limitations. Based on our investigations, we propose a simple yet effective approach to adapt CLIP for supervised object Re-ID. Our approach directly fine-tunes the image encoder of CLIP using a prototypical contrastive learning (PCL) loss, eliminating the need for prompt learning. Experimental results on both person and vehicle Re-ID datasets demonstrate the competitiveness of our method compared to CLIP-ReID. Furthermore, we extend our PCL-based CLIP fine-tuning approach to unsupervised scenarios, where we achieve state-of-the art performance. Code is available at https://github.com/RikoLi/PCL-CLIP.