SurfaceXR: Fusing Smartwatch IMUs and Egocentric Hand Pose for Seamless Surface Interactions

TL;DR

SurfaceXR combines smartwatch IMU data with headset-based hand tracking to improve surface interaction accuracy and comfort in XR, overcoming challenges of hand tracking and surface estimation.

Contribution

This paper introduces a novel sensor fusion method that integrates smartwatch IMUs with egocentric hand tracking for enhanced surface interaction in XR.

Findings

Significant improvement in touch tracking accuracy.

Enhanced 8-class gesture recognition performance.

Validated effectiveness through a 21-participant study.

Abstract

Mid-air gestures in Extended Reality (XR) often cause fatigue and imprecision. Surface-based interactions offer improved accuracy and comfort, but current egocentric vision methods struggle due to hand tracking challenges and unreliable surface plane estimation. We introduce SurfaceXR, a sensor fusion approach combining headset-based hand tracking with smartwatch IMU data to enable robust inputs on everyday surfaces. Our insight is that these modalities are complementary: hand tracking provides 3D positional data while IMUs capture high-frequency motion. A 21-participant study validates SurfaceXR's effectiveness for touch tracking and 8-class gesture recognition, demonstrating significant improvements over single-modality approaches.