Human Activity Recognition in RGB-D Videos by Dynamic Images

TL;DR

This paper introduces a novel approach for recognizing human activities in challenging RGB-D videos with multiple performers by using dynamic images and deep neural networks, achieving superior results on new and existing datasets.

Contribution

The paper presents a new method that captures motion information through dynamic images from RGB-D videos with multiple performers, a task not previously addressed.

Findings

Achieved comparable performance on MSR Action 3D dataset.

Outperformed state-of-the-art on a new multi-performer dataset.

Effectively extracted motion features reducing complexity.

Abstract

Human Activity Recognition in RGB-D videos has been an active research topic during the last decade. However, no efforts have been found in the literature, for recognizing human activity in RGB-D videos where several performers are performing simultaneously. In this paper we introduce such a challenging dataset with several performers performing the activities. We present a novel method for recognizing human activities in such videos. The proposed method aims in capturing the motion information of the whole video by producing a dynamic image corresponding to the input video. We use two parallel ResNext-101 to produce the dynamic images for the RGB video and depth video separately. The dynamic images contain only the motion information and hence, the unnecessary background information are eliminated. We send the two dynamic images extracted from the RGB and Depth videos respectively, through a fully connected layer of neural networks. The proposed dynamic image reduces the complexity of the recognition process by extracting a sparse matrix from a video. However, the proposed system maintains the required motion information for recognizing the activity. The proposed method has been tested on the MSR Action 3D dataset and has shown comparable performances with respect to the state-of-the-art. We also apply the proposed method on our own dataset, where the proposed method outperforms the state-of-the-art approaches.