GMMFormer v2: An Uncertainty-aware Framework for Partially Relevant Video Retrieval

TL;DR

GMMFormer v2 introduces an uncertainty-aware framework for partially relevant video retrieval, enhancing clip modeling and text-clip matching to improve accuracy and efficiency in retrieving relevant video segments without moment annotations.

Contribution

It proposes novel temporal consolidation and matching losses to address uncertainty and semantic collapse in PRVR, outperforming previous methods.

Findings

Significant performance improvements on three PRVR benchmarks.

Effective uncertainty modeling enhances clip perception and matching accuracy.

Versatile text-clip matching reduces semantic collapse.

Abstract

Given a text query, partially relevant video retrieval (PRVR) aims to retrieve untrimmed videos containing relevant moments. Due to the lack of moment annotations, the uncertainty lying in clip modeling and text-clip correspondence leads to major challenges. Despite the great progress, existing solutions either sacrifice efficiency or efficacy to capture varying and uncertain video moments. What's worse, few methods have paid attention to the text-clip matching pattern under such uncertainty, exposing the risk of semantic collapse. To address these issues, we present GMMFormer v2, an uncertainty-aware framework for PRVR. For clip modeling, we improve a strong baseline GMMFormer with a novel temporal consolidation module upon multi-scale contextual features, which maintains efficiency and improves the perception for varying moments. To achieve uncertainty-aware text-clip matching, we upgrade the query diverse loss in GMMFormer to facilitate fine-grained uniformity and propose a novel optimal matching loss for fine-grained text-clip alignment. Their collaboration alleviates the semantic collapse phenomenon and neatly promotes accurate correspondence between texts and moments. We conduct extensive experiments and ablation studies on three PRVR benchmarks, demonstrating remarkable improvement of GMMFormer v2 compared to the past SOTA competitor and the versatility of uncertainty-aware text-clip matching for PRVR. Code is available at \url{https://github.com/huangmozhi9527/GMMFormer_v2}.