Deep Convolutional Neural Networks for Action Recognition Using Depth Map Sequences

TL;DR

This paper introduces a novel deep learning framework using Hierarchical Depth Motion Maps and 3-channel ConvNets for human action recognition from depth sequences, achieving state-of-the-art results across multiple datasets.

Contribution

It proposes a new view-invariant method combining HDMM and multi-channel ConvNets for improved action recognition from depth data.

Findings

Achieves state-of-the-art accuracy on multiple datasets.

Handles view variation effectively through 3D rotation.

Maintains performance on combined large datasets.

Abstract

Recently, deep learning approach has achieved promising results in various fields of computer vision. In this paper, a new framework called Hierarchical Depth Motion Maps (HDMM) + 3 Channel Deep Convolutional Neural Networks (3ConvNets) is proposed for human action recognition using depth map sequences. Firstly, we rotate the original depth data in 3D pointclouds to mimic the rotation of cameras, so that our algorithms can handle view variant cases. Secondly, in order to effectively extract the body shape and motion information, we generate weighted depth motion maps (DMM) at several temporal scales, referred to as Hierarchical Depth Motion Maps (HDMM). Then, three channels of ConvNets are trained on the HDMMs from three projected orthogonal planes separately. The proposed algorithms are evaluated on MSRAction3D, MSRAction3DExt, UTKinect-Action and MSRDailyActivity3D datasets respectively. We also combine the last three datasets into a larger one (called Combined Dataset) and test the proposed method on it. The results show that our approach can achieve state-of-the-art results on the individual datasets and without dramatical performance degradation on the Combined Dataset.