HAN: An Efficient Hierarchical Self-Attention Network for Skeleton-Based Gesture Recognition

TL;DR

This paper introduces HAN, a hierarchical self-attention network that efficiently captures spatial and temporal features for skeleton-based gesture recognition without using CNNs or GCNs, achieving competitive results with lower complexity.

Contribution

The paper proposes a novel pure self-attention based hierarchical network for gesture recognition, effectively modeling hand joint features without CNNs or GCNs.

Findings

Achieves competitive accuracy on three datasets.

Reduces computational complexity compared to existing methods.

Effectively captures local and global features of hand gestures.

Abstract

Previous methods for skeleton-based gesture recognition mostly arrange the skeleton sequence into a pseudo picture or spatial-temporal graph and apply deep Convolutional Neural Network (CNN) or Graph Convolutional Network (GCN) for feature extraction. Although achieving superior results, these methods have inherent limitations in dynamically capturing local features of interactive hand parts, and the computing efficiency still remains a serious issue. In this work, the self-attention mechanism is introduced to alleviate this problem. Considering the hierarchical structure of hand joints, we propose an efficient hierarchical self-attention network (HAN) for skeleton-based gesture recognition, which is based on pure self-attention without any CNN, RNN or GCN operators. Specifically, the joint self-attention module is used to capture spatial features of fingers, the finger self-attention module is designed to aggregate features of the whole hand. In terms of temporal features, the temporal self-attention module is utilized to capture the temporal dynamics of the fingers and the entire hand. Finally, these features are fused by the fusion self-attention module for gesture classification. Experiments show that our method achieves competitive results on three gesture recognition datasets with much lower computational complexity.