TSM: Temporal Shift Module for Efficient and Scalable Video Understanding on Edge Device

TL;DR

The paper introduces the Temporal Shift Module (TSM), a novel method that enhances 2D CNNs with temporal modeling capabilities, achieving high accuracy and efficiency for video understanding on edge devices.

Contribution

TSM is a simple, effective module that enables temporal information exchange in 2D CNNs without extra computation or parameters, improving scalability and performance.

Findings

TSM ranks first on the Something-Something leaderboard.

Achieves 74fps on Jetson Nano and 29fps on Galaxy Note8.

Enables large-scale training on 1,536 GPUs in 15 minutes.

Abstract

The explosive growth in video streaming requires video understanding at high accuracy and low computation cost. Conventional 2D CNNs are computationally cheap but cannot capture temporal relationships; 3D CNN-based methods can achieve good performance but are computationally intensive. In this paper, we propose a generic and effective Temporal Shift Module (TSM) that enjoys both high efficiency and high performance. The key idea of TSM is to shift part of the channels along the temporal dimension, thus facilitate information exchanged among neighboring frames. It can be inserted into 2D CNNs to achieve temporal modeling at zero computation and zero parameters. TSM offers several unique advantages. Firstly, TSM has high performance; it ranks the first on the Something-Something leaderboard upon submission. Secondly, TSM has high efficiency; it achieves a high frame rate of 74fps and 29fps for online video recognition on Jetson Nano and Galaxy Note8. Thirdly, TSM has higher scalability compared to 3D networks, enabling large-scale Kinetics training on 1,536 GPUs in 15 minutes. Lastly, TSM enables action concepts learning, which 2D networks cannot model; we visualize the category attention map and find that spatial-temporal action detector emerges during the training of classification tasks. The code is publicly available at https://github.com/mit-han-lab/temporal-shift-module.