GTM: Gray Temporal Model for Video Recognition

TL;DR

This paper introduces a gray stream input modality for video recognition, which simplifies data processing and enhances spatio-temporal modeling without additional computation, validated on multiple benchmarks.

Contribution

Proposes a new gray stream modality and a 1D-ICSC module for efficient, effective video action recognition with zero computation and parameters.

Findings

Achieves state-of-the-art results on Kinetics, Something-Something, HMDB-51, UCF-101

Demonstrates improved spatio-temporal modeling with zero computation

Validates effectiveness and efficiency of the proposed methods

Abstract

Data input modality plays an important role in video action recognition. Normally, there are three types of input: RGB, flow stream and compressed data. In this paper, we proposed a new input modality: gray stream. Specifically, taken the stacked consecutive 3 gray images as input, which is the same size of RGB, can not only skip the conversion process from video decoding data to RGB, but also improve the spatio-temporal modeling ability at zero computation and zero parameters. Meanwhile, we proposed a 1D Identity Channel-wise Spatio-temporal Convolution(1D-ICSC) which captures the temporal relationship at channel-feature level within a controllable computation budget(by parameters G & R). Finally, we confirm its effectiveness and efficiency on several action recognition benchmarks, such as Kinetics, Something-Something, HMDB-51 and UCF-101, and achieve impressive results.