Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learning

TL;DR

This paper introduces Hi-TRS, a hierarchical Transformer-based model that employs self-supervised pre-training to effectively learn spatial and temporal features of human skeleton sequences, improving performance across multiple downstream tasks.

Contribution

It proposes a novel hierarchical self-supervised pre-training scheme integrated into a Transformer encoder to explicitly model multi-level dependencies in skeleton sequences.

Findings

Achieves state-of-the-art results on action recognition, detection, and motion prediction.

Demonstrates strong transferability of learned representations across tasks.

Outperforms existing contrastive learning methods in skeleton sequence modeling.

Abstract

Despite the success of fully-supervised human skeleton sequence modeling, utilizing self-supervised pre-training for skeleton sequence representation learning has been an active field because acquiring task-specific skeleton annotations at large scales is difficult. Recent studies focus on learning video-level temporal and discriminative information using contrastive learning, but overlook the hierarchical spatial-temporal nature of human skeletons. Different from such superficial supervision at the video level, we propose a self-supervised hierarchical pre-training scheme incorporated into a hierarchical Transformer-based skeleton sequence encoder (Hi-TRS), to explicitly capture spatial, short-term, and long-term temporal dependencies at frame, clip, and video levels, respectively. To evaluate the proposed self-supervised pre-training scheme with Hi-TRS, we conduct extensive experiments covering three skeleton-based downstream tasks including action recognition, action detection, and motion prediction. Under both supervised and semi-supervised evaluation protocols, our method achieves the state-of-the-art performance. Additionally, we demonstrate that the prior knowledge learned by our model in the pre-training stage has strong transfer capability for different downstream tasks.