Vid-Morp: Video Moment Retrieval Pretraining from Unlabeled Videos in the Wild

TL;DR

Vid-Morp introduces a novel pretraining approach for video moment retrieval using unlabeled videos and pseudo annotations, significantly reducing annotation costs and achieving strong zero-shot and unsupervised performance.

Contribution

The paper proposes Vid-Morp, a large-scale unlabeled video dataset and the ReCorrect algorithm for effective pretraining without manual annotations.

Findings

ReCorrect achieves over 75% of fully-supervised performance in zero-shot settings.

Unsupervised ReCorrect reaches about 85% performance on benchmarks.

Pretraining with pseudo labels reduces annotation costs significantly.

Abstract

Given a natural language query, video moment retrieval aims to localize the described temporal moment in an untrimmed video. A major challenge of this task is its heavy dependence on labor-intensive annotations for training. Unlike existing works that directly train models on manually curated data, we propose a novel paradigm to reduce annotation costs: pretraining the model on unlabeled, real-world videos. To support this, we introduce Video Moment Retrieval Pretraining (Vid-Morp), a large-scale dataset collected with minimal human intervention, consisting of over 50K videos captured in the wild and 200K pseudo annotations. Direct pretraining on these imperfect pseudo annotations, however, presents significant challenges, including mismatched sentence-video pairs and imprecise temporal boundaries. To address these issues, we propose the ReCorrect algorithm, which comprises two main phases: semantics-guided refinement and memory-consensus correction. The semantics-guided refinement enhances the pseudo labels by leveraging semantic similarity with video frames to clean out unpaired data and make initial adjustments to temporal boundaries. In the following memory-consensus correction phase, a memory bank tracks the model predictions, progressively correcting the temporal boundaries based on consensus within the memory. Comprehensive experiments demonstrate ReCorrect's strong generalization abilities across multiple downstream settings. Zero-shot ReCorrect achieves over 75% and 80% of the best fully-supervised performance on two benchmarks, while unsupervised ReCorrect reaches about 85% on both. The code, dataset, and pretrained models are available at https://github.com/baopj/Vid-Morp.