Aligning Videos in Space and Time

TL;DR

This paper introduces a novel method for aligning videos in space and time by learning cross-video cycle-consistency, enabling accurate correspondence of semantically similar patches across videos.

Contribution

It proposes a new alignment procedure that leverages cycle-consistency for learning spatial and temporal correspondences without extensive labeled data.

Findings

Successfully aligns patches across videos in experiments

Learns representations sensitive to object and action states

Demonstrates effectiveness on Penn Action and Pouring datasets

Abstract

In this paper, we focus on the task of extracting visual correspondences across videos. Given a query video clip from an action class, we aim to align it with training videos in space and time. Obtaining training data for such a fine-grained alignment task is challenging and often ambiguous. Hence, we propose a novel alignment procedure that learns such correspondence in space and time via cross video cycle-consistency. During training, given a pair of videos, we compute cycles that connect patches in a given frame in the first video by matching through frames in the second video. Cycles that connect overlapping patches together are encouraged to score higher than cycles that connect non-overlapping patches. Our experiments on the Penn Action and Pouring datasets demonstrate that the proposed method can successfully learn to correspond semantically similar patches across videos, and learns representations that are sensitive to object and action states.