Sync from the Sea: Retrieving Alignable Videos from Large-Scale Datasets

TL;DR

This paper introduces a new task called Alignable Video Retrieval (AVR), which identifies and synchronizes videos from large datasets based on their alignability to a query video, enhancing video analysis and editing capabilities.

Contribution

It proposes DRAQ for identifying alignable videos, develops a robust frame-level feature design, and establishes a benchmark with evaluation protocols for AVR.

Findings

Effective identification of alignable videos across datasets

Improved alignment performance with proposed features

Successful large-scale experiments demonstrating approach's robustness

Abstract

Temporal video alignment aims to synchronize the key events like object interactions or action phase transitions in two videos. Such methods could benefit various video editing, processing, and understanding tasks. However, existing approaches operate under the restrictive assumption that a suitable video pair for alignment is given, significantly limiting their broader applicability. To address this, we re-pose temporal alignment as a search problem and introduce the task of Alignable Video Retrieval (AVR). Given a query video, our approach can identify well-alignable videos from a large collection of clips and temporally synchronize them to the query. To achieve this, we make three key contributions: 1) we introduce DRAQ, a video alignability indicator to identify and re-rank the best alignable video from a set of candidates; 2) we propose an effective and generalizable frame-level video feature design to improve the alignment performance of several off-the-shelf feature representations, and 3) we propose a novel benchmark and evaluation protocol for AVR using cycle-consistency metrics. Our experiments on 3 datasets, including large-scale Kinetics700, demonstrate the effectiveness of our approach in identifying alignable video pairs from diverse datasets. Project Page: https://daveishan.github.io/avr-webpage/.