CLIP-based Synergistic Knowledge Transfer for Text-based Person Retrieval

TL;DR

This paper introduces a CLIP-based method for text-based person retrieval that effectively transfers knowledge between vision and language modalities, achieving superior performance with minimal additional training parameters.

Contribution

The paper proposes a novel synergistic knowledge transfer framework using bidirectional prompts and dual adapters, enhancing CLIP's capabilities for TPR with efficient parameter usage.

Findings

Outperforms state-of-the-art on three benchmarks

Uses only 7.4% of total model parameters for training

Demonstrates high efficiency, effectiveness, and generalization

Abstract

Text-based Person Retrieval (TPR) aims to retrieve the target person images given a textual query. The primary challenge lies in bridging the substantial gap between vision and language modalities, especially when dealing with limited large-scale datasets. In this paper, we introduce a CLIP-based Synergistic Knowledge Transfer (CSKT) approach for TPR. Specifically, to explore the CLIP's knowledge on input side, we first propose a Bidirectional Prompts Transferring (BPT) module constructed by text-to-image and image-to-text bidirectional prompts and coupling projections. Secondly, Dual Adapters Transferring (DAT) is designed to transfer knowledge on output side of Multi-Head Attention (MHA) in vision and language. This synergistic two-way collaborative mechanism promotes the early-stage feature fusion and efficiently exploits the existing knowledge of CLIP. CSKT outperforms the state-of-the-art approaches across three benchmark datasets when the training parameters merely account for 7.4% of the entire model, demonstrating its remarkable efficiency, effectiveness and generalization.