SkeleTR: Towrads Skeleton-based Action Recognition in the Wild

TL;DR

SkeleTR introduces a two-stage skeleton-based action recognition framework that effectively models intra-person dynamics and inter-person interactions, achieving state-of-the-art results in diverse scenarios.

Contribution

It proposes a novel two-stage approach combining graph convolutions and Transformer encoders for flexible, multi-task skeleton-based action recognition in unconstrained environments.

Findings

Achieves state-of-the-art performance on multiple benchmarks.

Effectively handles variable numbers of people and interactions.

Improves transfer learning and joint training across tasks.

Abstract

We present SkeleTR, a new framework for skeleton-based action recognition. In contrast to prior work, which focuses mainly on controlled environments, we target more general scenarios that typically involve a variable number of people and various forms of interaction between people. SkeleTR works with a two-stage paradigm. It first models the intra-person skeleton dynamics for each skeleton sequence with graph convolutions, and then uses stacked Transformer encoders to capture person interactions that are important for action recognition in general scenarios. To mitigate the negative impact of inaccurate skeleton associations, SkeleTR takes relative short skeleton sequences as input and increases the number of sequences. As a unified solution, SkeleTR can be directly applied to multiple skeleton-based action tasks, including video-level action classification, instance-level action detection, and group-level activity recognition. It also enables transfer learning and joint training across different action tasks and datasets, which result in performance improvement. When evaluated on various skeleton-based action recognition benchmarks, SkeleTR achieves the state-of-the-art performance.