Elevating Skeleton-Based Action Recognition with Efficient Multi-Modality Self-Supervision

TL;DR

This paper introduces a novel framework for skeleton-based action recognition that leverages multi-modality data through an Implicit Knowledge Exchange Module and a relational distillation approach, improving accuracy and efficiency.

Contribution

It proposes an Implicit Knowledge Exchange Module and a relational cross-modality knowledge distillation framework to enhance multi-modality skeleton-based action recognition.

Findings

Improved recognition accuracy with multi-modality data.

Effective knowledge propagation control between modalities.

Enhanced efficiency with the teacher-student distillation framework.

Abstract

Self-supervised representation learning for human action recognition has developed rapidly in recent years. Most of the existing works are based on skeleton data while using a multi-modality setup. These works overlooked the differences in performance among modalities, which led to the propagation of erroneous knowledge between modalities while only three fundamental modalities, i.e., joints, bones, and motions are used, hence no additional modalities are explored. In this work, we first propose an Implicit Knowledge Exchange Module (IKEM) which alleviates the propagation of erroneous knowledge between low-performance modalities. Then, we further propose three new modalities to enrich the complementary information between modalities. Finally, to maintain efficiency when introducing new modalities, we propose a novel teacher-student framework to distill the knowledge from the secondary modalities into the mandatory modalities considering the relationship constrained by anchors, positives, and negatives, named relational cross-modality knowledge distillation. The experimental results demonstrate the effectiveness of our approach, unlocking the efficient use of skeleton-based multi-modality data. Source code will be made publicly available at https://github.com/desehuileng0o0/IKEM.