Spatial Hierarchy and Temporal Attention Guided Cross Masking for Self-supervised Skeleton-based Action Recognition

TL;DR

This paper proposes HA-CM, a novel self-supervised skeleton-based action recognition framework that employs hierarchical and attention-guided cross-masking in both spatial and temporal domains to improve feature learning and robustness.

Contribution

The paper introduces a hierarchical and attention-guided cross-masking framework (HA-CM) that applies novel masking strategies in spatial and temporal aspects for skeleton sequences.

Findings

Effective on NTU-60, NTU-120, and PKU-MMD datasets.

Improves model robustness and feature learning.

Outperforms existing self-supervised methods.

Abstract

In self-supervised skeleton-based action recognition, the mask reconstruction paradigm is gaining interest in enhancing model refinement and robustness through effective masking. However, previous works primarily relied on a single masking criterion, resulting in the model overfitting specific features and overlooking other effective information. In this paper, we introduce a hierarchy and attention guided cross-masking framework (HA-CM) that applies masking to skeleton sequences from both spatial and temporal perspectives. Specifically, in spatial graphs, we utilize hyperbolic space to maintain joint distinctions and effectively preserve the hierarchical structure of high-dimensional skeletons, employing joint hierarchy as the masking criterion. In temporal flows, we substitute traditional distance metrics with the global attention of joints for masking, addressing the convergence of distances in high-dimensional space and the lack of a global perspective. Additionally, we incorporate cross-contrast loss based on the cross-masking framework into the loss function to enhance the model's learning of instance-level features. HA-CM shows efficiency and universality on three public large-scale datasets, NTU-60, NTU-120, and PKU-MMD. The source code of our HA-CM is available at https://github.com/YinxPeng/HA-CM-main.