T2VParser: Adaptive Decomposition Tokens for Partial Alignment in Text to Video Retrieval

TL;DR

T2VParser introduces adaptive semantic decomposition tokens to improve partial alignment in text-to-video retrieval, effectively handling the richer and partial content of videos compared to images.

Contribution

The paper proposes Adaptive Decomposition Tokens for extracting multiview semantic representations, enabling precise partial alignment in text-video retrieval tasks.

Findings

Achieves accurate partial alignment in experiments

Effectively decomposes cross-modal content

Retains pretrained model knowledge

Abstract

Text-to-video retrieval essentially aims to train models to align visual content with textual descriptions accurately. Due to the impressive general multimodal knowledge demonstrated by image-text pretrained models such as CLIP, existing work has primarily focused on extending CLIP knowledge for video-text tasks. However, videos typically contain richer information than images. In current video-text datasets, textual descriptions can only reflect a portion of the video content, leading to partial misalignment in video-text matching. Therefore, directly aligning text representations with video representations can result in incorrect supervision, ignoring the inequivalence of information. In this work, we propose T2VParser to extract multiview semantic representations from text and video, achieving adaptive semantic alignment rather than aligning the entire representation. To extract corresponding representations from different modalities, we introduce Adaptive Decomposition Tokens, which consist of a set of learnable tokens shared across modalities. The goal of T2VParser is to emphasize precise alignment between text and video while retaining the knowledge of pretrained models. Experimental results demonstrate that T2VParser achieves accurate partial alignment through effective cross-modal content decomposition. The code is available at https://github.com/Lilidamowang/T2VParser.