Multi-proposal Collaboration and Multi-task Training for Weakly-supervised Video Moment Retrieval

TL;DR

This paper introduces MCMT, a novel weakly-supervised approach for Video Moment Retrieval that generates multiple proposals, uses Gaussian masks, and employs multi-task training for improved accuracy and stability.

Contribution

The paper proposes a new method combining multi-proposal generation, Gaussian masks, and multi-task training to enhance weakly-supervised VMR performance and stability.

Findings

Outperforms existing methods on standard benchmarks

Produces high-quality temporal proposals and stable retrieval results

Effectively distinguishes relevant video segments without temporal annotations

Abstract

This study focuses on weakly-supervised Video Moment Retrieval (VMR), aiming to identify a moment semantically similar to the given query within an untrimmed video using only video-level correspondences, without relying on temporal annotations during training. Previous methods either aggregate predictions for all instances in the video, or indirectly address the task by proposing reconstructions for the query. However, these methods often produce low-quality temporal proposals, struggle with distinguishing misaligned moments in the same video, or lack stability due to a reliance on a single auxiliary task. To address these limitations, we present a novel weakly-supervised method called Multi-proposal Collaboration and Multi-task Training (MCMT). Initially, we generate multiple proposals and derive corresponding learnable Gaussian masks from them. These masks are then combined to create a high-quality positive sample mask, highlighting video clips most relevant to the query. Concurrently, we classify other clips in the same video as the easy negative sample and the entire video as the hard negative sample. During training, we introduce forward and inverse masked query reconstruction tasks to impose more substantial constraints on the network, promoting more robust and stable retrieval performance. Extensive experiments on two standard benchmarks affirm the effectiveness of the proposed method in VMR.