Beyond-Voice: Towards Continuous 3D Hand Pose Tracking on Commercial Home Assistant Devices

TL;DR

Beyond-Voice introduces a novel acoustic sensing system that enables commodity home assistants to continuously track 3D hand poses with high accuracy using existing microphones and speakers, enhancing accessibility without additional hardware.

Contribution

It presents a deep learning-based acoustic hand tracking system that operates across environments without personalized training, achieving high-fidelity 3D finger joint reconstruction.

Findings

Average joint tracking error of 16.47mm

Operates across different environments and users

No personalized training data needed

Abstract

The surging popularity of home assistants and their voice user interface (VUI) have made them an ideal central control hub for smart home devices. However, current form factors heavily rely on VUI, which poses accessibility and usability issues; some latest ones are equipped with additional cameras and displays, which are costly and raise privacy concerns. These concerns jointly motivate Beyond-Voice, a novel high-fidelity acoustic sensing system that allows commodity home assistant devices to track and reconstruct hand poses continuously. It transforms the home assistant into an active sonar system using its existing onboard microphones and speakers. We feed a high-resolution range profile to the deep learning model that can analyze the motions of multiple body parts and predict the 3D positions of 21 finger joints, bringing the granularity for acoustic hand tracking to the next level. It operates across different environments and users without the need for personalized training data. A user study with 11 participants in 3 different environments shows that Beyond-Voice can track joints with an average mean absolute error of 16.47mm without any training data provided by the testing subject.