Multi-Scale Spatial-Temporal Self-Attention Graph Convolutional Networks for Skeleton-based Action Recognition

TL;DR

This paper introduces MSST-GCN, a hybrid graph convolutional network utilizing multi-scale spatial-temporal self-attention to enhance skeleton-based action recognition by capturing complex intra-frame and inter-frame dependencies.

Contribution

The paper proposes a novel multi-scale spatial-temporal self-attention GCN model that improves modeling of long-range dependencies in skeleton-based action recognition tasks.

Findings

Achieved state-of-the-art results on multiple datasets.

Effectively models long-range spatial and temporal dependencies.

Enhances intra-frame and inter-frame interaction understanding.

Abstract

Skeleton-based gesture recognition methods have achieved high success using Graph Convolutional Network (GCN). In addition, context-dependent adaptive topology as a neighborhood vertex information and attention mechanism leverages a model to better represent actions. In this paper, we propose self-attention GCN hybrid model, Multi-Scale Spatial-Temporal self-attention (MSST)-GCN to effectively improve modeling ability to achieve state-of-the-art results on several datasets. We utilize spatial self-attention module with adaptive topology to understand intra-frame interactions within a frame among different body parts, and temporal self-attention module to examine correlations between frames of a node. These two are followed by multi-scale convolution network with dilations, which not only captures the long-range temporal dependencies of joints but also the long-range spatial dependencies (i.e., long-distance dependencies) of node temporal behaviors. They are combined into high-level spatial-temporal representations and output the predicted action with the softmax classifier.