Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networks

TL;DR

This paper introduces a deep LSTM network with a novel regularization scheme and dropout algorithm for skeleton-based action recognition, effectively capturing joint co-occurrences and modeling long-term dependencies.

Contribution

It presents an end-to-end deep LSTM model with a new regularization and dropout method specifically designed for learning joint co-occurrence features in skeleton data.

Findings

Effective on three human action datasets

Improves recognition accuracy

Captures joint co-occurrence patterns

Abstract

Skeleton based action recognition distinguishes human actions using the trajectories of skeleton joints, which provide a very good representation for describing actions. Considering that recurrent neural networks (RNNs) with Long Short-Term Memory (LSTM) can learn feature representations and model long-term temporal dependencies automatically, we propose an end-to-end fully connected deep LSTM network for skeleton based action recognition. Inspired by the observation that the co-occurrences of the joints intrinsically characterize human actions, we take the skeleton as the input at each time slot and introduce a novel regularization scheme to learn the co-occurrence features of skeleton joints. To train the deep LSTM network effectively, we propose a new dropout algorithm which simultaneously operates on the gates, cells, and output responses of the LSTM neurons. Experimental results on three human action recognition datasets consistently demonstrate the effectiveness of the proposed model.