Severe Domain Shift in Skeleton-Based Action Recognition:A Study of Uncertainty Failure in Real-World Gym Environments

TL;DR

This paper investigates the severe domain shift challenges in skeleton-based action recognition when transitioning from controlled to real-world gym environments, highlighting the limitations of current uncertainty detection methods and proposing a gating mechanism for safer deployment.

Contribution

The study introduces a novel Gym2D dataset, systematically analyzes the failure of uncertainty methods under domain shift, and proposes a lightweight gating mechanism to improve safety in real-world applications.

Findings

Skeleton Transformer accuracy drops from 63.2% to below 2% under domain shift.

High OOD detection AUROC does not correlate with safe decision-making.

Gating mechanism significantly reduces confident wrong predictions.

Abstract

The practical deployment gap -- transitioning from controlled multi-view 3D skeleton capture to unconstrained monocular 2D pose estimation -- introduces a compound domain shift whose safety implications remain critically underexplored. We present a systematic study of this severe domain shift using a novel Gym2D dataset (style/viewpoint shift) and the UCF101 dataset (semantic shift). Our Skeleton Transformer achieves 63.2% cross-subject accuracy on NTU-120 but drops to 1.6% under zero-shot transfer to the Gym domain and 1.16% on UCF101. Critically, we demonstrate that high Out-Of-Distribution (OOD) detection AUROC does not guarantee safe selective classification. Standard uncertainty methods fail to detect this performance drop: the model remains confidently incorrect with 99.6% risk even at 50% coverage across both OOD datasets. While energy-based scoring (AUROC >= 0.91) and Mahalanobis distance provide reliable distributional detection signals, such high AUROC scores coexist with poor risk-coverage behavior when making decisions. A lightweight finetuned gating mechanism restores calibration and enables graceful abstention, substantially reducing the rate of confident wrong predictions. Our work challenges standard deployment assumptions, providing a principled safety analysis of both semantic and geometric skeleton recognition deployment.