OCHID-Fi: Occlusion-Robust Hand Pose Estimation in 3D via RF-Vision

TL;DR

OCHID-Fi introduces a novel RF-vision based 3D hand pose estimation method that effectively handles occlusions, surpassing traditional camera-based approaches by leveraging wideband RF sensors and advanced training techniques.

Contribution

It is the first RF-HPE method capable of 3D hand pose estimation behind obstacles, using cross-modality training and adversarial learning for occlusion robustness.

Findings

Achieves comparable accuracy to camera-based methods in unobstructed scenarios.

Maintains high accuracy even with occlusions, demonstrating robustness.

Shows strong generalization to new domains and unseen occluded scenarios.

Abstract

Hand Pose Estimation (HPE) is crucial to many applications, but conventional cameras-based CM-HPE methods are completely subject to Line-of-Sight (LoS), as cameras cannot capture occluded objects. In this paper, we propose to exploit Radio-Frequency-Vision (RF-vision) capable of bypassing obstacles for achieving occluded HPE, and we introduce OCHID-Fi as the first RF-HPE method with 3D pose estimation capability. OCHID-Fi employs wideband RF sensors widely available on smart devices (e.g., iPhones) to probe 3D human hand pose and extract their skeletons behind obstacles. To overcome the challenge in labeling RF imaging given its human incomprehensible nature, OCHID-Fi employs a cross-modality and cross-domain training process. It uses a pre-trained CM-HPE network and a synchronized CM/RF dataset, to guide the training of its complex-valued RF-HPE network under LoS conditions. It further transfers knowledge learned from labeled LoS domain to unlabeled occluded domain via adversarial learning, enabling OCHID-Fi to generalize to unseen occluded scenarios. Experimental results demonstrate the superiority of OCHID-Fi: it achieves comparable accuracy to CM-HPE under normal conditions while maintaining such accuracy even in occluded scenarios, with empirical evidence for its generalizability to new domains.