Global Temporal Representation based CNNs for Infrared Action Recognition

TL;DR

This paper introduces a novel global temporal representation called OFSDI for infrared action recognition, combining local, global, and spatial temporal features extracted via CNNs and aggregated with LLC and SVM for improved accuracy.

Contribution

It proposes a new global temporal feature representation (OFSDI) and a multi-stream CNN framework with feature aggregation and LLC for infrared action recognition.

Findings

Outperforms state-of-the-art methods on InfAR and NTU RGB+D datasets.

Effective integration of local, global, and spatial temporal features.

Enhanced robustness of features using LLC.

Abstract

Infrared human action recognition has many advantages, i.e., it is insensitive to illumination change, appearance variability, and shadows. Existing methods for infrared action recognition are either based on spatial or local temporal information, however, the global temporal information, which can better describe the movements of body parts across the whole video, is not considered. In this letter, we propose a novel global temporal representation named optical-flow stacked difference image (OFSDI) and extract robust and discriminative feature from the infrared action data by considering the local, global, and spatial temporal information together. Due to the small size of the infrared action dataset, we first apply convolutional neural networks on local, spatial, and global temporal stream respectively to obtain efficient convolutional feature maps from the raw data rather than train a classifier directly. Then these convolutional feature maps are aggregated into effective descriptors named three-stream trajectory-pooled deep-convolutional descriptors by trajectory-constrained pooling. Furthermore, we improve the robustness of these features by using the locality-constrained linear coding (LLC) method. With these features, a linear support vector machine (SVM) is adopted to classify the action data in our scheme. We conduct the experiments on infrared action recognition datasets InfAR and NTU RGB+D. The experimental results show that the proposed approach outperforms the representative state-of-the-art handcrafted features and deep learning features based methods for the infrared action recognition.