Augmenting Moment Retrieval: Zero-Dependency Two-Stage Learning

TL;DR

This paper introduces AMR, a zero-dependency two-stage learning framework for moment retrieval that enhances boundary and semantic discrimination without additional data, significantly improving performance on benchmarks.

Contribution

It proposes a novel zero-external-dependency augmentation method and a two-stage training framework with curriculum learning and distillation for better moment retrieval.

Findings

AMR outperforms previous state-of-the-art methods on multiple benchmarks.

The augmentation effectively resolves boundary ambiguity and semantic confusion.

The two-stage training improves boundary and semantic understanding.

Abstract

Existing Moment Retrieval methods face three critical bottlenecks: (1) data scarcity forces models into shallow keyword-feature associations; (2) boundary ambiguity in transition regions between adjacent events; (3) insufficient discrimination of fine-grained semantics (e.g., distinguishing ``kicking" vs. ``throwing" a ball). In this paper, we propose a zero-external-dependency Augmented Moment Retrieval framework, AMR, designed to overcome local optima caused by insufficient data annotations and the lack of robust boundary and semantic discrimination capabilities. AMR is built upon two key insights: (1) it resolves ambiguous boundary information and semantic confusion in existing annotations without additional data (avoiding costly manual labeling), and (2) it preserves boundary and semantic discriminative capabilities enhanced by training while generalizing to real-world scenarios, significantly improving performance. Furthermore, we propose a two-stage training framework with cold-start and distillation adaptation. The cold-start stage employs curriculum learning on augmented data to build foundational boundary/semantic awareness. The distillation stage introduces dual query sets: Original Queries maintain DETR-based localization using frozen Base Queries from the cold-start model, while Active Queries dynamically adapt to real-data distributions. A cross-stage distillation loss enforces consistency between Original and Base Queries, preventing knowledge forgetting while enabling real-world generalization. Experiments on multiple benchmarks show that AMR achieves improved performance over prior state-of-the-art approaches.