Visuo-Acoustic Hand Pose and Contact Estimation

TL;DR

VibeMesh is a wearable visuo-acoustic system that combines vision and active acoustic sensing with a graph neural network to accurately estimate hand pose and contact points, especially under occlusion.

Contribution

It introduces a novel, non-intrusive visuo-acoustic platform and a cross-modal graph network for dense hand pose and contact estimation, along with a new dataset.

Findings

Outperforms vision-only methods in accuracy

Robust in occluded and static-contact scenarios

Provides dense, high-resolution contact predictions

Abstract

Accurately estimating hand pose and hand-object contact events is essential for robot data-collection, immersive virtual environments, and biomechanical analysis, yet remains challenging due to visual occlusion, subtle contact cues, limitations in vision-only sensing, and the lack of accessible and flexible tactile sensing. We therefore introduce VibeMesh, a novel wearable system that fuses vision with active acoustic sensing for dense, per-vertex hand contact and pose estimation. VibeMesh integrates a bone-conduction speaker and sparse piezoelectric microphones, distributed on a human hand, emitting structured acoustic signals and capturing their propagation to infer changes induced by contact. To interpret these cross-modal signals, we propose a graph-based attention network that processes synchronized audio spectra and RGB-D-derived hand meshes to predict contact with high spatial resolution. We contribute: (i) a lightweight, non-intrusive visuo-acoustic sensing platform; (ii) a cross-modal graph network for joint pose and contact inference; (iii) a dataset of synchronized RGB-D, acoustic, and ground-truth contact annotations across diverse manipulation scenarios; and (iv) empirical results showing that VibeMesh outperforms vision-only baselines in accuracy and robustness, particularly in occluded or static-contact settings.