Spatio-Temporal Tuples Transformer for Skeleton-Based Action Recognition

TL;DR

This paper introduces STTFormer, a novel transformer-based model that captures joint correlations across space and time in skeleton sequences, improving action recognition accuracy.

Contribution

The paper proposes a spatio-temporal tuples self-attention module and a feature aggregation module to better model joint relationships across frames, surpassing existing methods.

Findings

Achieves superior performance on large-scale datasets

Effectively models joint correlations across frames

Outperforms state-of-the-art methods in accuracy

Abstract

Capturing the dependencies between joints is critical in skeleton-based action recognition task. Transformer shows great potential to model the correlation of important joints. However, the existing Transformer-based methods cannot capture the correlation of different joints between frames, which the correlation is very useful since different body parts (such as the arms and legs in "long jump") between adjacent frames move together. Focus on this problem, A novel spatio-temporal tuples Transformer (STTFormer) method is proposed. The skeleton sequence is divided into several parts, and several consecutive frames contained in each part are encoded. And then a spatio-temporal tuples self-attention module is proposed to capture the relationship of different joints in consecutive frames. In addition, a feature aggregation module is introduced between non-adjacent frames to enhance the ability to distinguish similar actions. Compared with the state-of-the-art methods, our method achieves better performance on two large-scale datasets.