Adaptive Evidential Learning for Temporal-Semantic Robustness in Moment Retrieval

TL;DR

This paper introduces DEMR, a novel framework that improves temporal-semantic robustness in video moment retrieval by addressing modality imbalance and uncertainty estimation issues through cross-modal alignment, query reconstruction, and a Geom-regularizer.

Contribution

The paper proposes DEMR, a new method with a Reflective Flipped Fusion block, query reconstruction, and Geom-regularizer to enhance robustness and uncertainty estimation in moment retrieval.

Findings

Significant improvements on ActivityNet-CD and Charades-CD datasets.

Enhanced robustness and interpretability in moment retrieval.

Better uncertainty calibration and modality alignment.

Abstract

In the domain of moment retrieval, accurately identifying temporal segments within videos based on natural language queries remains challenging. Traditional methods often employ pre-trained models that struggle with fine-grained information and deterministic reasoning, leading to difficulties in aligning with complex or ambiguous moments. To overcome these limitations, we explore Deep Evidential Regression (DER) to construct a vanilla Evidential baseline. However, this approach encounters two major issues: the inability to effectively handle modality imbalance and the structural differences in DER's heuristic uncertainty regularizer, which adversely affect uncertainty estimation. This misalignment results in high uncertainty being incorrectly associated with accurate samples rather than challenging ones. Our observations indicate that existing methods lack the adaptability required for complex video scenarios. In response, we propose Debiased Evidential Learning for Moment Retrieval (DEMR), a novel framework that incorporates a Reflective Flipped Fusion (RFF) block for cross-modal alignment and a query reconstruction task to enhance text sensitivity, thereby reducing bias in uncertainty estimation. Additionally, we introduce a Geom-regularizer to refine uncertainty predictions, enabling adaptive alignment with difficult moments and improving retrieval accuracy. Extensive testing on standard datasets and debiased datasets ActivityNet-CD and Charades-CD demonstrates significant enhancements in effectiveness, robustness, and interpretability, positioning our approach as a promising solution for temporal-semantic robustness in moment retrieval. The code is publicly available at https://github.com/KaijingOfficial/DEMR.