Moment of Untruth: Dealing with Negative Queries in Video Moment Retrieval

TL;DR

This paper introduces Negative-Aware Video Moment Retrieval (NA-VMR), a new task that evaluates both moment localization and negative query rejection, proposing a new benchmark and a model adaptation that improves negative rejection accuracy.

Contribution

It defines NA-VMR with new benchmarks, analyzes current models' limitations, and proposes UniVTG-NA, significantly improving negative rejection while maintaining retrieval performance.

Findings

UniVTG-NA achieves 98.4% negative rejection accuracy.

Current SOTA models struggle with negative query rejection.

New benchmarks for in-domain and out-of-domain negatives are provided.

Abstract

Video Moment Retrieval is a common task to evaluate the performance of visual-language models - it involves localising start and end times of moments in videos from query sentences. The current task formulation assumes that the queried moment is present in the video, resulting in false positive moment predictions when irrelevant query sentences are provided. In this paper we propose the task of Negative-Aware Video Moment Retrieval (NA-VMR), which considers both moment retrieval accuracy and negative query rejection accuracy. We make the distinction between In-Domain and Out-of-Domain negative queries and provide new evaluation benchmarks for two popular video moment retrieval datasets: QVHighlights and Charades-STA. We analyse the ability of current SOTA video moment retrieval approaches to adapt to Negative-Aware Video Moment Retrieval and propose UniVTG-NA, an adaptation of UniVTG designed to tackle NA-VMR. UniVTG-NA achieves high negative rejection accuracy (avg. $98.4\%$) scores while retaining moment retrieval scores to within $3.87\%$ Recall@1. Dataset splits and code are available at https://github.com/keflanagan/MomentofUntruth