MaskSem: Semantic-Guided Masking for Learning 3D Hybrid High-Order Motion Representation

TL;DR

MaskSem introduces a semantic-guided masking strategy and hybrid high-order motion reconstruction to improve 3D skeleton-based action recognition, especially for complex motions in human-robot interaction.

Contribution

The paper proposes MaskSem, a novel framework that uses Grad-CAM guided masking and hybrid high-order motion targets to enhance self-supervised learning of complex motion patterns.

Findings

Improves recognition accuracy on NTU60, NTU120, and PKU-MMD datasets.

Enhances model's understanding of complex motion patterns.

Suitable for human-robot interaction applications.

Abstract

Human action recognition is a crucial task for intelligent robotics, particularly within the context of human-robot collaboration research. In self-supervised skeleton-based action recognition, the mask-based reconstruction paradigm learns the spatial structure and motion patterns of the skeleton by masking joints and reconstructing the target from unlabeled data. However, existing methods focus on a limited set of joints and low-order motion patterns, limiting the model's ability to understand complex motion patterns. To address this issue, we introduce MaskSem, a novel semantic-guided masking method for learning 3D hybrid high-order motion representations. This novel framework leverages Grad-CAM based on relative motion to guide the masking of joints, which can be represented as the most semantically rich temporal orgions. The semantic-guided masking process can encourage the model to explore more discriminative features. Furthermore, we propose using hybrid high-order motion as the reconstruction target, enabling the model to learn multi-order motion patterns. Specifically, low-order motion velocity and high-order motion acceleration are used together as the reconstruction target. This approach offers a more comprehensive description of the dynamic motion process, enhancing the model's understanding of motion patterns. Experiments on the NTU60, NTU120, and PKU-MMD datasets show that MaskSem, combined with a vanilla transformer, improves skeleton-based action recognition, making it more suitable for applications in human-robot interaction.