Enhancing Action Recognition from Low-Quality Skeleton Data via Part-Level Knowledge Distillation

TL;DR

This paper introduces a novel part-level knowledge distillation framework that significantly improves action recognition accuracy from low-quality skeleton data by leveraging high-quality skeletons and local part-based matching strategies.

Contribution

It proposes a new part-based skeleton matching and a part-level contrastive loss to enhance low-quality skeleton action recognition using high-quality data guidance.

Findings

Improved accuracy on NTU-RGB+D, Penn Action, SYSU 3D HOI datasets.

Effective knowledge transfer from high- to low-quality skeletons.

Robustness to missing or inaccurate joints in skeleton data.

Abstract

Skeleton-based action recognition is vital for comprehending human-centric videos and has applications in diverse domains. One of the challenges of skeleton-based action recognition is dealing with low-quality data, such as skeletons that have missing or inaccurate joints. This paper addresses the issue of enhancing action recognition using low-quality skeletons through a general knowledge distillation framework. The proposed framework employs a teacher-student model setup, where a teacher model trained on high-quality skeletons guides the learning of a student model that handles low-quality skeletons. To bridge the gap between heterogeneous high-quality and lowquality skeletons, we present a novel part-based skeleton matching strategy, which exploits shared body parts to facilitate local action pattern learning. An action-specific part matrix is developed to emphasize critical parts for different actions, enabling the student model to distill discriminative part-level knowledge. A novel part-level multi-sample contrastive loss achieves knowledge transfer from multiple high-quality skeletons to low-quality ones, which enables the proposed knowledge distillation framework to include training low-quality skeletons that lack corresponding high-quality matches. Comprehensive experiments conducted on the NTU-RGB+D, Penn Action, and SYSU 3D HOI datasets demonstrate the effectiveness of the proposed knowledge distillation framework.