Multi-scale 2D Representation Learning for weakly-supervised moment retrieval

TL;DR

This paper introduces a multi-scale 2D representation learning approach for weakly-supervised video moment retrieval, effectively capturing temporal dependencies across scales to improve retrieval accuracy without needing detailed annotations.

Contribution

It proposes a novel 2D map construction for temporal dependencies and a candidate selection method, advancing weakly-supervised video retrieval techniques.

Findings

Achieves superior performance on Charades-STA and ActivityNet Captions datasets.

Effectively models temporal dependencies across multiple scales.

Outperforms state-of-the-art weakly-supervised methods.

Abstract

Video moment retrieval aims to search the moment most relevant to a given language query. However, most existing methods in this community often require temporal boundary annotations which are expensive and time-consuming to label. Hence weakly supervised methods have been put forward recently by only using coarse video-level label. Despite effectiveness, these methods usually process moment candidates independently, while ignoring a critical issue that the natural temporal dependencies between candidates in different temporal scales. To cope with this issue, we propose a Multi-scale 2D Representation Learning method for weakly supervised video moment retrieval. Specifically, we first construct a two-dimensional map for each temporal scale to capture the temporal dependencies between candidates. Two dimensions in this map indicate the start and end time points of these candidates. Then, we select top-K candidates from each scale-varied map with a learnable convolutional neural network. With a newly designed Moments Evaluation Module, we obtain the alignment scores of the selected candidates. At last, the similarity between captions and language query is served as supervision for further training the candidates' selector. Experiments on two benchmark datasets Charades-STA and ActivityNet Captions demonstrate that our approach achieves superior performance to state-of-the-art results.