Skeleton-based Action Recognition with Non-linear Dependency Modeling and Hilbert-Schmidt Independence Criterion

TL;DR

This paper introduces a novel framework for skeleton-based action recognition that models non-linear dependencies between all joints and employs the Hilbert-Schmidt Independence Criterion to improve class differentiation, achieving state-of-the-art results.

Contribution

It proposes a dependency refinement method for modeling all joint dependencies and a Hilbert-Schmidt Independence Criterion-based framework for robust action classification.

Findings

Achieves state-of-the-art performance on NTU RGB+D datasets.

Effectively models non-linear joint dependencies beyond connected joints.

Utilizes independence criterion to handle high-dimensional motion data.

Abstract

Human skeleton-based action recognition has long been an indispensable aspect of artificial intelligence. Current state-of-the-art methods tend to consider only the dependencies between connected skeletal joints, limiting their ability to capture non-linear dependencies between physically distant joints. Moreover, most existing approaches distinguish action classes by estimating the probability density of motion representations, yet the high-dimensional nature of human motions invokes inherent difficulties in accomplishing such measurements. In this paper, we seek to tackle these challenges from two directions: (1) We propose a novel dependency refinement approach that explicitly models dependencies between any pair of joints, effectively transcending the limitations imposed by joint distance. (2) We further propose a framework that utilizes the Hilbert-Schmidt Independence Criterion to differentiate action classes without being affected by data dimensionality, and mathematically derive learning objectives guaranteeing precise recognition. Empirically, our approach sets the state-of-the-art performance on NTU RGB+D, NTU RGB+D 120, and Northwestern-UCLA datasets.