LSTA-Net: Long short-term Spatio-Temporal Aggregation Network for Skeleton-based Action Recognition

TL;DR

LSTA-Net is a novel architecture for skeleton-based action recognition that effectively captures long- and short-range spatio-temporal dependencies using a factorised design and attention mechanisms, outperforming existing methods.

Contribution

The paper introduces LSTA-Net, a new model that better models long- and short-range dependencies in skeleton sequences through a factorised architecture and channel-wise attention.

Findings

Outperforms state-of-the-art methods on benchmark datasets

Effectively captures long- and short-range dependencies in space and time

Uses a pure factorised architecture with attention mechanisms

Abstract

Modelling various spatio-temporal dependencies is the key to recognising human actions in skeleton sequences. Most existing methods excessively relied on the design of traversal rules or graph topologies to draw the dependencies of the dynamic joints, which is inadequate to reflect the relationships of the distant yet important joints. Furthermore, due to the locally adopted operations, the important long-range temporal information is therefore not well explored in existing works. To address this issue, in this work we propose LSTA-Net: a novel Long short-term Spatio-Temporal Aggregation Network, which can effectively capture the long/short-range dependencies in a spatio-temporal manner. We devise our model into a pure factorised architecture which can alternately perform spatial feature aggregation and temporal feature aggregation. To improve the feature aggregation effect, a channel-wise attention mechanism is also designed and employed. Extensive experiments were conducted on three public benchmark datasets, and the results suggest that our approach can capture both long-and-short range dependencies in the space and time domain, yielding higher results than other state-of-the-art methods. Code available at https://github.com/tailin1009/LSTA-Net.