Learning to Align Sequential Actions in the Wild

TL;DR

This paper introduces a novel method for aligning sequential actions in videos that accounts for temporal variations, background frames, and non-monotonic sequences, outperforming existing self-supervised approaches.

Contribution

The proposed approach enforces temporal priors on the optimal transport matrix, enabling alignment of diverse and non-monotonic action sequences in the wild.

Findings

Outperforms state-of-the-art on four benchmarks

Handles background frames effectively

Accounts for non-monotonic action sequences

Abstract

State-of-the-art methods for self-supervised sequential action alignment rely on deep networks that find correspondences across videos in time. They either learn frame-to-frame mapping across sequences, which does not leverage temporal information, or assume monotonic alignment between each video pair, which ignores variations in the order of actions. As such, these methods are not able to deal with common real-world scenarios that involve background frames or videos that contain non-monotonic sequence of actions. In this paper, we propose an approach to align sequential actions in the wild that involve diverse temporal variations. To this end, we propose an approach to enforce temporal priors on the optimal transport matrix, which leverages temporal consistency, while allowing for variations in the order of actions. Our model accounts for both monotonic and non-monotonic sequences and handles background frames that should not be aligned. We demonstrate that our approach consistently outperforms the state-of-the-art in self-supervised sequential action representation learning on four different benchmark datasets.