Collaborative Spatio-temporal Feature Learning for Video Action Recognition

TL;DR

This paper introduces a novel collaborative spatio-temporal feature learning method for video action recognition that shares weights across views, leading to improved performance and interpretability.

Contribution

The paper proposes a weight-sharing neural operation that encodes spatio-temporal features by performing 2D convolutions on orthogonal views, enhancing feature collaboration and interpretability.

Findings

Achieved state-of-the-art results on large-scale benchmarks.

Won 1st place in the Moments in Time Challenge 2018.

Provided insights into the contributions of spatial and temporal features.

Abstract

Spatio-temporal feature learning is of central importance for action recognition in videos. Existing deep neural network models either learn spatial and temporal features independently (C2D) or jointly with unconstrained parameters (C3D). In this paper, we propose a novel neural operation which encodes spatio-temporal features collaboratively by imposing a weight-sharing constraint on the learnable parameters. In particular, we perform 2D convolution along three orthogonal views of volumetric video data,which learns spatial appearance and temporal motion cues respectively. By sharing the convolution kernels of different views, spatial and temporal features are collaboratively learned and thus benefit from each other. The complementary features are subsequently fused by a weighted summation whose coefficients are learned end-to-end. Our approach achieves state-of-the-art performance on large-scale benchmarks and won the 1st place in the Moments in Time Challenge 2018. Moreover, based on the learned coefficients of different views, we are able to quantify the contributions of spatial and temporal features. This analysis sheds light on interpretability of the model and may also guide the future design of algorithm for video recognition.