MomentMix Augmentation with Length-Aware DETR for Temporally Robust Moment Retrieval

TL;DR

This paper introduces MomentMix augmentation and a length-aware decoder to improve the localization of short video moments in retrieval tasks, significantly enhancing performance of DETR-based models.

Contribution

It proposes MomentMix augmentation strategies and a length-aware decoder to address short moment localization challenges in video retrieval.

Findings

Outperforms state-of-the-art DETR-based methods on benchmark datasets.

Achieves 9.62% gain in R1@0.7 on QVHighlights.

Improves mAP by 16.9% on QVHighlights.

Abstract

Video Moment Retrieval (MR) aims to localize moments within a video based on a given natural language query. Given the prevalent use of platforms like YouTube for information retrieval, the demand for MR techniques is significantly growing. Recent DETR-based models have made notable advances in performance but still struggle with accurately localizing short moments. Through data analysis, we identified limited feature diversity in short moments, which motivated the development of MomentMix. MomentMix generates new short-moment samples by employing two augmentation strategies: ForegroundMix and BackgroundMix, each enhancing the ability to understand the query-relevant and irrelevant frames, respectively. Additionally, our analysis of prediction bias revealed that short moments particularly struggle with accurately predicting their center positions and length of moments. To address this, we propose a Length-Aware Decoder, which conditions length through a novel bipartite matching process. Our extensive studies demonstrate the efficacy of our length-aware approach, especially in localizing short moments, leading to improved overall performance. Our method surpasses state-of-the-art DETR-based methods on benchmark datasets, achieving the highest R1 and mAP on QVHighlights and the highest R1@0.7 on TACoS and Charades-STA (such as a 9.62% gain in R1@0.7 and an 16.9% gain in mAP average for QVHighlights). The code is available at https://github.com/sjpark5800/LA-DETR.