Event-aware Video Corpus Moment Retrieval

TL;DR

This paper introduces EventFormer, a novel model for Video Corpus Moment Retrieval that explicitly models events within videos, leading to improved accuracy and efficiency over existing frame-aware methods.

Contribution

EventFormer leverages event reasoning and hierarchical encoding to better capture semantic structures, achieving state-of-the-art results in VCMR benchmarks.

Findings

EventFormer outperforms previous methods on TVR, ANetCaps, and DiDeMo datasets.

The model effectively captures event-level information for improved retrieval accuracy.

EventFormer demonstrates efficiency and effectiveness in partially relevant video retrieval tasks.

Abstract

Video Corpus Moment Retrieval (VCMR) is a practical video retrieval task focused on identifying a specific moment within a vast corpus of untrimmed videos using the natural language query. Existing methods for VCMR typically rely on frame-aware video retrieval, calculating similarities between the query and video frames to rank videos based on maximum frame similarity.However, this approach overlooks the semantic structure embedded within the information between frames, namely, the event, a crucial element for human comprehension of videos. Motivated by this, we propose EventFormer, a model that explicitly utilizes events within videos as fundamental units for video retrieval. The model extracts event representations through event reasoning and hierarchical event encoding. The event reasoning module groups consecutive and visually similar frame representations into events, while the hierarchical event encoding encodes information at both the frame and event levels. We also introduce anchor multi-head self-attenion to encourage Transformer to capture the relevance of adjacent content in the video. The training of EventFormer is conducted by two-branch contrastive learning and dual optimization for two sub-tasks of VCMR. Extensive experiments on TVR, ANetCaps, and DiDeMo benchmarks show the effectiveness and efficiency of EventFormer in VCMR, achieving new state-of-the-art results. Additionally, the effectiveness of EventFormer is also validated on partially relevant video retrieval task.