2DP-2MRC: 2-Dimensional Pointer-based Machine Reading Comprehension Method for Multimodal Moment Retrieval

TL;DR

This paper introduces 2DP-2MRC, a novel multimodal moment retrieval method that improves localization accuracy while reducing computational costs by combining coarse-grained video encoding with a 2D pointer mechanism.

Contribution

The paper proposes a new 2D pointer-based model with an AV-Encoder for better moment localization and lower complexity compared to existing methods.

Findings

Outperforms baseline models on HiREST dataset

Achieves more accurate moment boundary detection

Reduces computational complexity in retrieval tasks

Abstract

Moment retrieval aims to locate the most relevant moment in an untrimmed video based on a given natural language query. Existing solutions can be roughly categorized into moment-based and clip-based methods. The former often involves heavy computations, while the latter, due to overlooking coarse-grained information, typically underperforms compared to moment-based models. Hence, this paper proposes a novel 2-Dimensional Pointer-based Machine Reading Comprehension for Moment Retrieval Choice (2DP-2MRC) model to address the issue of imprecise localization in clip-based methods while maintaining lower computational complexity than moment-based methods. Specifically, we introduce an AV-Encoder to capture coarse-grained information at moment and video levels. Additionally, a 2D pointer encoder module is introduced to further enhance boundary detection for target moment. Extensive experiments on the HiREST dataset demonstrate that 2DP-2MRC significantly outperforms existing baseline models.