Heterogeneous Skeleton-Based Action Representation Learning

TL;DR

This paper introduces a novel framework for learning unified action representations from heterogeneous skeleton data, addressing variations in joint structures and topologies to improve action recognition across diverse sources.

Contribution

It proposes a dual-component method that converts 2D to 3D skeletons, constructs skeleton-specific prompts, and incorporates semantic motion encoding for better heterogeneous skeleton processing.

Findings

Effective on NTU-60, NTU-120, and PKU-MMD II datasets

Improves action recognition accuracy across diverse skeleton sources

Applicable to robots with different humanoid structures

Abstract

Skeleton-based human action recognition has received widespread attention in recent years due to its diverse range of application scenarios. Due to the different sources of human skeletons, skeleton data naturally exhibit heterogeneity. The previous works, however, overlook the heterogeneity of human skeletons and solely construct models tailored for homogeneous skeletons. This work addresses the challenge of heterogeneous skeleton-based action representation learning, specifically focusing on processing skeleton data that varies in joint dimensions and topological structures. The proposed framework comprises two primary components: heterogeneous skeleton processing and unified representation learning. The former first converts two-dimensional skeleton data into three-dimensional skeleton via an auxiliary network, and then constructs a prompted unified skeleton using skeleton-specific prompts. We also design an additional modality named semantic motion encoding to harness the semantic information within skeletons. The latter module learns a unified action representation using a shared backbone network that processes different heterogeneous skeletons. Extensive experiments on the NTU-60, NTU-120, and PKU-MMD II datasets demonstrate the effectiveness of our method in various tasks of action understanding. Our approach can be applied to action recognition in robots with different humanoid structures.