Modal-specific Pseudo Query Generation for Video Corpus Moment Retrieval

TL;DR

This paper introduces MPGN, a self-supervised framework that generates pseudo queries from multimodal video data to localize moments without explicit annotations, improving video corpus moment retrieval.

Contribution

The novel MPGN framework enables moment localization in videos without relying on expensive annotations by generating pseudo queries from multimodal information.

Findings

Achieves competitive results on TVR dataset.

Effective in learning without explicit annotations.

Utilizes multimodal information for pseudo query generation.

Abstract

Video corpus moment retrieval (VCMR) is the task to retrieve the most relevant video moment from a large video corpus using a natural language query. For narrative videos, e.g., dramas or movies, the holistic understanding of temporal dynamics and multimodal reasoning is crucial. Previous works have shown promising results; however, they relied on the expensive query annotations for VCMR, i.e., the corresponding moment intervals. To overcome this problem, we propose a self-supervised learning framework: Modal-specific Pseudo Query Generation Network (MPGN). First, MPGN selects candidate temporal moments via subtitle-based moment sampling. Then, it generates pseudo queries exploiting both visual and textual information from the selected temporal moments. Through the multimodal information in the pseudo queries, we show that MPGN successfully learns to localize the video corpus moment without any explicit annotation. We validate the effectiveness of MPGN on the TVR dataset, showing competitive results compared with both supervised models and unsupervised setting models.