Focal and Global Spatial-Temporal Transformer for Skeleton-based Action Recognition

TL;DR

This paper introduces FG-STFormer, a novel transformer model for skeleton-based action recognition that emphasizes discriminative local joints and short-range temporal dynamics, outperforming existing methods on multiple benchmarks.

Contribution

The paper proposes a new focal and global spatial-temporal transformer with joint and body part coupling and dilated temporal convolution, enhancing local and short-range temporal modeling.

Findings

Outperforms existing transformer-based methods on NTU-60, NTU-120, and NW-UCLA datasets.

Effectively models local joints and short-term temporal dynamics.

Achieves state-of-the-art results compared to GCN-based methods.

Abstract

Despite great progress achieved by transformer in various vision tasks, it is still underexplored for skeleton-based action recognition with only a few attempts. Besides, these methods directly calculate the pair-wise global self-attention equally for all the joints in both the spatial and temporal dimensions, undervaluing the effect of discriminative local joints and the short-range temporal dynamics. In this work, we propose a novel Focal and Global Spatial-Temporal Transformer network (FG-STFormer), that is equipped with two key components: (1) FG-SFormer: focal joints and global parts coupling spatial transformer. It forces the network to focus on modelling correlations for both the learned discriminative spatial joints and human body parts respectively. The selective focal joints eliminate the negative effect of non-informative ones during accumulating the correlations. Meanwhile, the interactions between the focal joints and body parts are incorporated to enhance the spatial dependencies via mutual cross-attention. (2) FG-TFormer: focal and global temporal transformer. Dilated temporal convolution is integrated into the global self-attention mechanism to explicitly capture the local temporal motion patterns of joints or body parts, which is found to be vital important to make temporal transformer work. Extensive experimental results on three benchmarks, namely NTU-60, NTU-120 and NW-UCLA, show our FG-STFormer surpasses all existing transformer-based methods, and compares favourably with state-of-the art GCN-based methods.