SkateFormer: Skeletal-Temporal Transformer for Human Action Recognition

TL;DR

SkateFormer introduces a skeletal-temporal transformer that partitions joints and frames to efficiently capture key relations for improved human action recognition from skeleton data.

Contribution

It proposes a novel partition-based self-attention mechanism that models skeletal and temporal relations efficiently, surpassing existing GCN and transformer methods.

Findings

Outperforms recent state-of-the-art methods on benchmark datasets.

Effectively captures key skeletal-temporal relations with partitioned attention.

Reduces computational resources compared to full self-attention models.

Abstract

Skeleton-based action recognition, which classifies human actions based on the coordinates of joints and their connectivity within skeleton data, is widely utilized in various scenarios. While Graph Convolutional Networks (GCNs) have been proposed for skeleton data represented as graphs, they suffer from limited receptive fields constrained by joint connectivity. To address this limitation, recent advancements have introduced transformer-based methods. However, capturing correlations between all joints in all frames requires substantial memory resources. To alleviate this, we propose a novel approach called Skeletal-Temporal Transformer (SkateFormer) that partitions joints and frames based on different types of skeletal-temporal relation (Skate-Type) and performs skeletal-temporal self-attention (Skate-MSA) within each partition. We categorize the key skeletal-temporal relations for action recognition into a total of four distinct types. These types combine (i) two skeletal relation types based on physically neighboring and distant joints, and (ii) two temporal relation types based on neighboring and distant frames. Through this partition-specific attention strategy, our SkateFormer can selectively focus on key joints and frames crucial for action recognition in an action-adaptive manner with efficient computation. Extensive experiments on various benchmark datasets validate that our SkateFormer outperforms recent state-of-the-art methods.