Grab-n-Go: On-the-Go Microgesture Recognition with Objects in Hand

TL;DR

Grab-n-Go is a wearable device that uses active acoustic sensing and deep learning to recognize fine-grained hand microgestures while holding objects, enabling seamless interaction with occupied hands.

Contribution

It introduces the first wearable system capable of recognizing microgestures, grasping poses, and object geometries simultaneously using a single wristband.

Findings

Achieved 92.0% average recognition accuracy in user studies.

Successfully recognized 30 microgestures across various grasping poses.

Demonstrated robustness against deformable and challenging objects.

Abstract

As computing devices become increasingly integrated into daily life, there is a growing need for intuitive, always-available interaction methods, even when users' hands are occupied. In this paper, we introduce Grab-n-Go, the first wearable device that leverages active acoustic sensing to recognize subtle hand microgestures while holding various objects. Unlike prior systems that focus solely on free-hand gestures or basic hand-object activity recognition, Grab-n-Go simultaneously captures information about hand microgestures, grasping poses, and object geometries using a single wristband, enabling the recognition of fine-grained hand movements occurring within activities involving occupied hands. A deep learning framework processes these complex signals to identify 30 distinct microgestures, with 6 microgestures for each of the 5 grasping poses. In a user study with 10 participants and 25 everyday objects, Grab-n-Go achieved an average recognition accuracy of 92.0%. A follow-up study further validated Grab-n-Go's robustness against 10 more challenging, deformable objects. These results underscore the potential of Grab-n-Go to provide seamless, unobtrusive interactions without requiring modifications to existing objects. The complete dataset, comprising data from 18 participants performing 30 microgestures with 35 distinct objects, is publicly available at https://github.com/cjlisalee/Grab-n-Go_Data with the DOI: https://doi.org/10.7298/7kbd-vv75.