AVI-HT: Adaptive Vision-IMU Fusion for 3D Hand Tracking

TL;DR

AVI-HT introduces an adaptive fusion method combining visual data and IMU signals for accurate 3D hand tracking, especially under occlusion, using a novel attention mechanism and extensive real-world data.

Contribution

The paper proposes a new adaptive vision-IMU fusion approach with a cross-sensor attention mechanism and a large annotated dataset for improved 3D hand tracking.

Findings

AVI-HT reduces mean keypoint error by 16.1%.

Wrist-aligned variant reduces error by 24.2%.

Ablation studies show finger-specific IMU contributions.

Abstract

We present AVI-HT, an adaptive visual-IMU fusion approach for tracking 3D hand poses by jointly modeling the egocentric image with on-glove 6-DoF IMU signals. AVI-HT achieves significantly improved accuracy and availability, particularly in hand-object interaction (HOI) scenarios involving heavy visual occlusion. Two complementary ingredients underpin its success: (1) synchronized multi-modal training data pairing on-body vision-IMU sensor streams with ground-truth 3D hand poses from a motion-capture system, and (2) a cross-sensor deep attention mechanism that adaptively modulates the trust assigned to the vision and individual IMU sensors. To evaluate AVI-HT in real-world settings, we conduct extensive experiments on our DexGloveHOI dataset that consists of 100K+ pairwise vision-IMU samples with synchronized 3D annotated poses, in which users manipulate a variety of objects during daily tasks. We compare against multiple single- and multi-modal tracking approaches under two hand models (UmeTrack, MANO). The results show that AVI-HT reduces mean keypoint error by 16.1% and its wrist-aligned variant by 24.2% over the baselines. Ablation studies further reveal the per-finger contribution of IMU sensors across activity types, and the model's sensitivity to IMU noise and temporal misalignment in vision-IMU fusion.