Mamba-VMR: Multimodal Query Augmentation via Generated Videos for Precise Temporal Grounding

TL;DR

Mamba-VMR introduces a two-stage multimodal framework that leverages generated videos and subtitle context to improve the accuracy and efficiency of text-driven video moment retrieval in long sequences.

Contribution

The paper proposes a novel two-stage approach combining subtitle-guided auxiliary video generation and a multi-modal network for enhanced temporal grounding, addressing limitations of previous methods.

Findings

Significant improvement in recall on the TVR benchmark.

Reduced computational costs compared to existing models.

Effective integration of generated motion priors and subtitle context.

Abstract

Text-driven video moment retrieval (VMR) remains challenging due to limited capture of hidden temporal dynamics in untrimmed videos, leading to imprecise grounding in long sequences. Traditional methods rely on natural language queries (NLQs) or static image augmentations, overlooking motion sequences and suffering from high computational costs in Transformer-based architectures. Existing approaches fail to integrate subtitle contexts and generated temporal priors effectively, we therefore propose a novel two-stage framework for enhanced temporal grounding. In the first stage, LLM-guided subtitle matching identifies relevant textual cues from video subtitles, fused with the query to generate auxiliary short videos via text-to-video models, capturing implicit motion information as temporal priors. In the second stage, augmented queries are processed through a multi-modal controlled Mamba network, extending text-controlled selection with video-guided gating for efficient fusion of generated priors and long sequences while filtering noise. Our framework is agnostic to base retrieval models and widely applicable for multimodal VMR. Experimental evaluations on the TVR benchmark demonstrate significant improvements over state-of-the-art methods, including reduced computational overhead and higher recall in long-sequence grounding.