LocVTP: Video-Text Pre-training for Temporal Localization

TL;DR

LocVTP introduces a novel video-text pre-training framework specifically designed to improve temporal localization tasks, demonstrating state-of-the-art results across multiple datasets and tasks.

Contribution

The paper proposes a new localization-oriented pre-training method with fine-grained alignment and temporal reasoning enhancements, addressing limitations of existing VTP methods for localization tasks.

Findings

Achieves state-of-the-art performance on multiple datasets.

Effectively improves temporal localization accuracy.

Enhances transferability of video-text representations.

Abstract

Video-Text Pre-training (VTP) aims to learn transferable representations for various downstream tasks from large-scale web videos. To date, almost all existing VTP methods are limited to retrieval-based downstream tasks, e.g., video retrieval, whereas their transfer potentials on localization-based tasks, e.g., temporal grounding, are under-explored. In this paper, we experimentally analyze and demonstrate the incompatibility of current VTP methods with localization tasks, and propose a novel Localization-oriented Video-Text Pre-training framework, dubbed as LocVTP. Specifically, we perform the fine-grained contrastive alignment as a complement to the coarse-grained one by a clip-word correspondence discovery scheme. To further enhance the temporal reasoning ability of the learned feature, we propose a context projection head and a temporal aware contrastive loss to perceive the contextual relationships. Extensive experiments on four downstream tasks across six datasets demonstrate that our LocVTP achieves state-of-the-art performance on both retrieval-based and localization-based tasks. Furthermore, we conduct comprehensive ablation studies and thorough analyses to explore the optimum model designs and training strategies.