Uneven Event Modeling for Partially Relevant Video Retrieval

TL;DR

This paper introduces an uneven event modeling framework for partially relevant video retrieval, improving event boundary detection and text-video alignment, leading to state-of-the-art results on benchmark datasets.

Contribution

The paper proposes a novel uneven event modeling approach with PGVS and CAER modules, enhancing event segmentation and representation for better retrieval accuracy.

Findings

Achieves state-of-the-art performance on PRVR benchmarks.

Effectively models uneven event boundaries using PGVS.

Refines event representations with context-aware CAER.

Abstract

Given a text query, partially relevant video retrieval (PRVR) aims to retrieve untrimmed videos containing relevant moments, wherein event modeling is crucial for partitioning the video into smaller temporal events that partially correspond to the text. Previous methods typically segment videos into a fixed number of equal-length clips, resulting in ambiguous event boundaries. Additionally, they rely on mean pooling to compute event representations, inevitably introducing undesired misalignment. To address these, we propose an Uneven Event Modeling (UEM) framework for PRVR. We first introduce the Progressive-Grouped Video Segmentation (PGVS) module, to iteratively formulate events in light of both temporal dependencies and semantic similarity between consecutive frames, enabling clear event boundaries. Furthermore, we also propose the Context-Aware Event Refinement (CAER) module to refine the event representation conditioned the text's cross-attention. This enables event representations to focus on the most relevant frames for a given text, facilitating more precise text-video alignment. Extensive experiments demonstrate that our method achieves state-of-the-art performance on two PRVR benchmarks. Code is available at https://github.com/Sasa77777779/UEM.git.