Learning Self-Similarity in Space and Time as Generalized Motion for Video Action Recognition

TL;DR

This paper introduces a novel spatio-temporal self-similarity (STSS) based motion representation for video action recognition, improving the modeling of motion dynamics and outperforming previous methods on multiple benchmarks.

Contribution

The paper proposes the SELFY neural block that captures long-term and fast motions using STSS, which can be integrated into existing architectures for end-to-end training.

Findings

Achieves state-of-the-art results on Something-Something-V1 & V2, Diving-48, and FineGym datasets.

Effectively models long-term interactions and fast motions in videos.

Demonstrates superiority and complementarity over previous motion modeling methods.

Abstract

Spatio-temporal convolution often fails to learn motion dynamics in videos and thus an effective motion representation is required for video understanding in the wild. In this paper, we propose a rich and robust motion representation based on spatio-temporal self-similarity (STSS). Given a sequence of frames, STSS represents each local region as similarities to its neighbors in space and time. By converting appearance features into relational values, it enables the learner to better recognize structural patterns in space and time. We leverage the whole volume of STSS and let our model learn to extract an effective motion representation from it. The proposed neural block, dubbed SELFY, can be easily inserted into neural architectures and trained end-to-end without additional supervision. With a sufficient volume of the neighborhood in space and time, it effectively captures long-term interaction and fast motion in the video, leading to robust action recognition. Our experimental analysis demonstrates its superiority over previous methods for motion modeling as well as its complementarity to spatio-temporal features from direct convolution. On the standard action recognition benchmarks, Something-Something-V1 & V2, Diving-48, and FineGym, the proposed method achieves the state-of-the-art results.