Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks

TL;DR

This paper introduces GCA-LSTM, a novel attention-based LSTM network that selectively focuses on informative skeletal joints using global context, significantly improving skeleton-based human action recognition accuracy.

Contribution

The paper proposes GCA-LSTM with a global context memory and recurrent attention mechanism, advancing the ability of LSTMs to focus on relevant joints for better recognition performance.

Findings

Achieves state-of-the-art results on five benchmark datasets.

Effectively models joint importance with global context-aware attention.

Improves recognition accuracy by focusing on informative joints.

Abstract

Human action recognition in 3D skeleton sequences has attracted a lot of research attention. Recently, Long Short-Term Memory (LSTM) networks have shown promising performance in this task due to their strengths in modeling the dependencies and dynamics in sequential data. As not all skeletal joints are informative for action recognition, and the irrelevant joints often bring noise which can degrade the performance, we need to pay more attention to the informative ones. However, the original LSTM network does not have explicit attention ability. In this paper, we propose a new class of LSTM network, Global Context-Aware Attention LSTM (GCA-LSTM), for skeleton based action recognition. This network is capable of selectively focusing on the informative joints in each frame of each skeleton sequence by using a global context memory cell. To further improve the attention capability of our network, we also introduce a recurrent attention mechanism, with which the attention performance of the network can be enhanced progressively. Moreover, we propose a stepwise training scheme in order to train our network effectively. Our approach achieves state-of-the-art performance on five challenging benchmark datasets for skeleton based action recognition.