A Flexible and Scalable Framework for Video Moment Search

TL;DR

This paper presents SPR, a scalable, flexible framework for video moment search that efficiently retrieves and refines relevant video segments to match text queries, outperforming existing methods in accuracy and speed.

Contribution

The paper introduces a novel three-stage framework, Segment-Proposal-Ranking, that improves scalability, flexibility, and efficiency in video moment retrieval tasks.

Findings

Achieves state-of-the-art performance on TVR-Ranking dataset.

Reduces computational cost and processing time significantly.

Allows independent improvements to each stage for adaptability.

Abstract

Video moment search, the process of finding relevant moments in a video corpus to match a user's query, is crucial for various applications. Existing solutions, however, often assume a single perfect matching moment, struggle with inefficient inference, and have limitations with hour-long videos. This paper introduces a flexible and scalable framework for retrieving a ranked list of moments from collection of videos in any length to match a text query, a task termed Ranked Video Moment Retrieval (RVMR). Our framework, called Segment-Proposal-Ranking (SPR), simplifies the search process into three independent stages: segment retrieval, proposal generation, and moment refinement with re-ranking. Specifically, videos are divided into equal-length segments with precomputed embeddings indexed offline, allowing efficient retrieval regardless of video length. For scalable online retrieval, both segments and queries are projected into a shared feature space to enable approximate nearest neighbor (ANN) search. Retrieved segments are then merged into coarse-grained moment proposals. Then a refinement and re-ranking module is designed to reorder and adjust timestamps of the coarse-grained proposals. Evaluations on the TVR-Ranking dataset demonstrate that our framework achieves state-of-the-art performance with significant reductions in computational cost and processing time. The flexible design also allows for independent improvements to each stage, making SPR highly adaptable for large-scale applications.