Combining Vision and EMG-Based Hand Tracking for Extended Reality Musical Instruments

TL;DR

This paper introduces a multimodal hand tracking system for XR musical instruments that combines vision and surface electromyography data to overcome self-occlusion issues, improving accuracy in finger joint estimation.

Contribution

The study presents a novel multimodal approach integrating vision and sEMG data for more accurate hand tracking in XR environments, especially under self-occlusion conditions.

Findings

Significant improvement in finger joint tracking accuracy.

Enhanced robustness of hand tracking during occlusion.

Potential for more natural XR musical instrument interactions.

Abstract

Hand tracking is a critical component of natural user interactions in extended reality (XR) environments, including extended reality musical instruments (XRMIs). However, self-occlusion remains a significant challenge for vision-based hand tracking systems, leading to inaccurate results and degraded user experiences. In this paper, we propose a multimodal hand tracking system that combines vision-based hand tracking with surface electromyography (sEMG) data for finger joint angle estimation. We validate the effectiveness of our system through a series of hand pose tasks designed to cover a wide range of gestures, including those prone to self-occlusion. By comparing the performance of our multimodal system to a baseline vision-based tracking method, we demonstrate that our multimodal approach significantly improves tracking accuracy for several finger joints prone to self-occlusion. These findings suggest that our system has the potential to enhance XR experiences by providing more accurate and robust hand tracking, even in the presence of self-occlusion.