WristSonic: Enabling Fine-grained Hand-Face Interactions on Smartwatches Using Active Acoustic Sensing

TL;DR

WristSonic is a wrist-worn acoustic sensing system that accurately detects fine-grained hand-face interactions using ultrasonic reflections and a Transformer neural network, enabling privacy-conscious gesture recognition in daily settings.

Contribution

It introduces a novel ultrasonic sensing approach combined with deep learning for precise, low-power detection of diverse hand-face gestures on wearables.

Findings

Achieves 93.08% macro F1-score in controlled settings.

Achieves 82.65% macro F1-score in semi-in-the-wild settings.

Detects 21 distinct hand-face actions with high accuracy.

Abstract

Hand-face interactions play a key role in many everyday tasks, providing insights into user habits, behaviors, intentions, and expressions. However, existing wearable sensing systems often struggle to track these interactions in daily settings due to their reliance on multiple sensors or privacy-sensitive, vision-based approaches. To address these challenges, we propose WristSonic, a wrist-worn active acoustic sensing system that uses speakers and microphones to capture ultrasonic reflections from hand, arm, and face movements, enabling fine-grained detection of hand-face interactions with minimal intrusion. By transmitting and analyzing ultrasonic waves, WristSonic distinguishes a wide range of gestures, such as tapping the temple, brushing teeth, and nodding, using a Transformer-based neural network architecture. This approach achieves robust recognition of 21 distinct actions with a single, low-power, privacy-conscious wearable. Through two user studies with 15 participants in controlled and semi-in-the-wild settings, WristSonic demonstrates high efficacy, achieving macro F1-scores of 93.08% and 82.65%, respectively.