Large-scale Isolated Gesture Recognition Using Convolutional Neural Networks

TL;DR

This paper introduces three novel depth sequence representations for gesture recognition, enabling effective use of ConvNets trained on image data, and achieves high accuracy on a large-scale challenge.

Contribution

It proposes three compact, effective depth sequence representations and demonstrates their use with ConvNets for improved gesture recognition accuracy.

Findings

Achieved 55.57% classification accuracy on LAP challenge

Ranked 2nd in the LAP 2016 challenge

Method performs well using only depth data

Abstract

This paper proposes three simple, compact yet effective representations of depth sequences, referred to respectively as Dynamic Depth Images (DDI), Dynamic Depth Normal Images (DDNI) and Dynamic Depth Motion Normal Images (DDMNI). These dynamic images are constructed from a sequence of depth maps using bidirectional rank pooling to effectively capture the spatial-temporal information. Such image-based representations enable us to fine-tune the existing ConvNets models trained on image data for classification of depth sequences, without introducing large parameters to learn. Upon the proposed representations, a convolutional Neural networks (ConvNets) based method is developed for gesture recognition and evaluated on the Large-scale Isolated Gesture Recognition at the ChaLearn Looking at People (LAP) challenge 2016. The method achieved 55.57\% classification accuracy and ranked $2^{nd}$ place in this challenge but was very close to the best performance even though we only used depth data.