When One Moment Isn't Enough: Multi-Moment Retrieval with Cross-Moment Interactions

TL;DR

This paper introduces a new multi-moment retrieval dataset and a novel framework, FlashMMR, to better handle real-world video temporal grounding where multiple relevant moments exist per query.

Contribution

The paper presents QV-M$^2$, a high-quality dataset for multi-moment retrieval, and proposes FlashMMR, a framework with post-verification for improved multi-moment video grounding.

Findings

QV-M$^2$ effectively benchmarks multi-moment retrieval.

FlashMMR outperforms previous methods on QV-M$^2$.

Retraining existing methods improves their performance in multi-moment scenarios.

Abstract

Existing Moment retrieval (MR) methods focus on Single-Moment Retrieval (SMR). However, one query can correspond to multiple relevant moments in real-world applications. This makes the existing datasets and methods insufficient for video temporal grounding. By revisiting the gap between current MR tasks and real-world applications, we introduce a high-quality datasets called QVHighlights Multi-Moment Dataset (QV-M$^2$), along with new evaluation metrics tailored for multi-moment retrieval (MMR). QV-M$^2$ consists of 2,212 annotations covering 6,384 video segments. Building on existing efforts in MMR, we propose a framework called FlashMMR. Specifically, we propose a Multi-moment Post-verification module to refine the moment boundaries. We introduce constrained temporal adjustment and subsequently leverage a verification module to re-evaluate the candidate segments. Through this sophisticated filtering pipeline, low-confidence proposals are pruned, and robust multi-moment alignment is achieved. We retrain and evaluate 6 existing MR methods on QV-M$^2$ and QVHighlights under both SMR and MMR settings. Results show that QV-M$^2$ serves as an effective benchmark for training and evaluating MMR models, while FlashMMR provides a strong baseline. Specifically, on QV-M$^2$, it achieves improvements over prior SOTA method by 3.00% on G-mAP, 2.70% on mAP@3+tgt, and 2.56% on mR@3. The proposed benchmark and method establish a foundation for advancing research in more realistic and challenging video temporal grounding scenarios. Code is released at https://github.com/Zhuo-Cao/QV-M2.