StNet: Local and Global Spatial-Temporal Modeling for Action Recognition

TL;DR

StNet introduces a novel architecture for action recognition in videos that combines local and global spatial-temporal modeling using super-images and a new temporal Xception block, outperforming existing methods.

Contribution

The paper proposes a new spatial-temporal network architecture that stacks frames into super-images and employs a novel temporal Xception block for improved video action recognition.

Findings

Outperforms several state-of-the-art methods on Kinetics dataset.

Achieves a good balance between accuracy and model complexity.

Generalizes well to UCF101 dataset.

Abstract

Despite the success of deep learning for static image understanding, it remains unclear what are the most effective network architectures for the spatial-temporal modeling in videos. In this paper, in contrast to the existing CNN+RNN or pure 3D convolution based approaches, we explore a novel spatial temporal network (StNet) architecture for both local and global spatial-temporal modeling in videos. Particularly, StNet stacks N successive video frames into a \emph{super-image} which has 3N channels and applies 2D convolution on super-images to capture local spatial-temporal relationship. To model global spatial-temporal relationship, we apply temporal convolution on the local spatial-temporal feature maps. Specifically, a novel temporal Xception block is proposed in StNet. It employs a separate channel-wise and temporal-wise convolution over the feature sequence of video. Extensive experiments on the Kinetics dataset demonstrate that our framework outperforms several state-of-the-art approaches in action recognition and can strike a satisfying trade-off between recognition accuracy and model complexity. We further demonstrate the generalization performance of the leaned video representations on the UCF101 dataset.