STSM: Spatio-Temporal Shift Module for Efficient Action Recognition

TL;DR

The paper introduces a Spatio-temporal Shift Module (STSM) that enhances 2D CNNs for action recognition by capturing temporal features efficiently without increasing computational costs, achieving state-of-the-art results.

Contribution

The novel STSM module is a plug-and-play component that improves 2D CNNs for video action recognition by modeling spatio-temporal features efficiently.

Findings

Achieved state-of-the-art results on Kinetics-400.

Improved network performance without increasing parameters.

Validated effectiveness across multiple datasets.

Abstract

The modeling, computational cost, and accuracy of traditional Spatio-temporal networks are the three most concentrated research topics in video action recognition. The traditional 2D convolution has a low computational cost, but it cannot capture the time relationship; the convolutional neural networks (CNNs) model based on 3D convolution can obtain good performance, but its computational cost is high, and the amount of parameters is large. In this paper, we propose a plug-and-play Spatio-temporal Shift Module (STSM), which is a generic module that is both effective and high-performance. Specifically, after STSM is inserted into other networks, the performance of the network can be improved without increasing the number of calculations and parameters. In particular, when the network is 2D CNNs, our STSM module allows the network to learn efficient Spatio-temporal features. We conducted extensive evaluations of the proposed module, conducted numerous experiments to study its effectiveness in video action recognition, and achieved state-of-the-art results on the kinetics-400 and Something-Something V2 datasets.