Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrieval

TL;DR

This paper introduces a novel method called Prompt Switch that enhances CLIP's video representations by incorporating spatial-temporal semantics through a prompt cube and captioning training, enabling efficient large-scale text-video retrieval.

Contribution

The paper proposes a new approach to adapt CLIP for video retrieval by using a prompt cube and captioning loss, avoiding complex fusion and enabling offline computation of video features.

Findings

Achieves state-of-the-art results on MSR-VTT, MSVD, and LSMDC datasets.

Efficiently incorporates global video semantics into frame representations.

Outperforms existing methods in large-scale retrieval scenarios.

Abstract

In text-video retrieval, recent works have benefited from the powerful learning capabilities of pre-trained text-image foundation models (e.g., CLIP) by adapting them to the video domain. A critical problem for them is how to effectively capture the rich semantics inside the video using the image encoder of CLIP. To tackle this, state-of-the-art methods adopt complex cross-modal modeling techniques to fuse the text information into video frame representations, which, however, incurs severe efficiency issues in large-scale retrieval systems as the video representations must be recomputed online for every text query. In this paper, we discard this problematic cross-modal fusion process and aim to learn semantically-enhanced representations purely from the video, so that the video representations can be computed offline and reused for different texts. Concretely, we first introduce a spatial-temporal "Prompt Cube" into the CLIP image encoder and iteratively switch it within the encoder layers to efficiently incorporate the global video semantics into frame representations. We then propose to apply an auxiliary video captioning objective to train the frame representations, which facilitates the learning of detailed video semantics by providing fine-grained guidance in the semantic space. With a naive temporal fusion strategy (i.e., mean-pooling) on the enhanced frame representations, we obtain state-of-the-art performances on three benchmark datasets, i.e., MSR-VTT, MSVD, and LSMDC.