BHaRNet: Reliability-Aware Body-Hand Modality Expertized Networks for Fine-grained Skeleton Action Recognition

TL;DR

This paper introduces BHaRNet, a novel reliability-aware dual-stream framework for fine-grained skeleton action recognition that effectively integrates multiple modalities and models uncertainty, leading to improved robustness and accuracy.

Contribution

It proposes a probabilistic dual-stream architecture with a calibration-free preprocessing, Noisy-OR fusion, and multi-modal ensemble, advancing skeleton-based HAR with uncertainty modeling and cross-modal integration.

Findings

Consistent performance improvements across multiple benchmarks.

Enhanced robustness under noisy and heterogeneous conditions.

Effective integration of structural and visual motion cues.

Abstract

Skeleton-based human action recognition (HAR) has achieved remarkable progress with graph-based architectures. However, most existing methods remain body-centric, focusing on large-scale motions while neglecting subtle hand articulations that are crucial for fine-grained recognition. This work presents a probabilistic dual-stream framework that unifies reliability modeling and multi-modal integration, generalizing expertized learning under uncertainty across both intra-skeleton and cross-modal domains. The framework comprises three key components: (1) a calibration-free preprocessing pipeline that removes canonical-space transformations and learns directly from native coordinates; (2) a probabilistic Noisy-OR fusion that stabilizes reliability-aware dual-stream learning without requiring explicit confidence supervision; and (3) an intra- to cross-modal ensemble that couples four skeleton modalities (Joint, Bone, Joint Motion, and Bone Motion) to RGB representations, bridging structural and visual motion cues in a unified cross-modal formulation. Comprehensive evaluations across multiple benchmarks (NTU RGB+D~60/120, PKU-MMD, N-UCLA) and a newly defined hand-centric benchmark exhibit consistent improvements and robustness under noisy and heterogeneous conditions.