Selective Query-guided Debiasing for Video Corpus Moment Retrieval

TL;DR

This paper introduces SQuiDNet, a novel video moment retrieval model that selectively debiases retrieval predictions by leveraging query semantics, improving accuracy and interpretability on multiple benchmarks.

Contribution

The paper proposes a new selective debiasing approach that preserves helpful biases while removing harmful ones, enhancing multi-modal retrieval performance.

Findings

Outperforms existing methods on TVR, ActivityNet, DiDeMo benchmarks.

Improves interpretability of retrieval results.

Effectively balances bias removal and preservation.

Abstract

Video moment retrieval (VMR) aims to localize target moments in untrimmed videos pertinent to a given textual query. Existing retrieval systems tend to rely on retrieval bias as a shortcut and thus, fail to sufficiently learn multi-modal interactions between query and video. This retrieval bias stems from learning frequent co-occurrence patterns between query and moments, which spuriously correlate objects (e.g., a pencil) referred in the query with moments (e.g., scene of writing with a pencil) where the objects frequently appear in the video, such that they converge into biased moment predictions. Although recent debiasing methods have focused on removing this retrieval bias, we argue that these biased predictions sometimes should be preserved because there are many queries where biased predictions are rather helpful. To conjugate this retrieval bias, we propose a Selective Query-guided Debiasing network (SQuiDNet), which incorporates the following two main properties: (1) Biased Moment Retrieval that intentionally uncovers the biased moments inherent in objects of the query and (2) Selective Query-guided Debiasing that performs selective debiasing guided by the meaning of the query. Our experimental results on three moment retrieval benchmarks (i.e., TVR, ActivityNet, DiDeMo) show the effectiveness of SQuiDNet and qualitative analysis shows improved interpretability.