Enabling hand gesture customization on wrist-worn devices
Xuhai Xu, Jun Gong, Carolina Brum, Lilian Liang, Bongsoo Suh, Kumar, Gupta, Yash Agarwal, Laurence Lindsey, Runchang Kang, Behrooz Shahsavari, Tu, Nguyen, Heriberto Nieto, Scott E. Hudson, Charlie Maalouf, Seyed Mousavi,, Gierad Laput

TL;DR
This paper introduces a lightweight, on-device gesture customization framework for wrist-worn devices, enabling users to add new gestures with minimal examples without affecting existing gesture recognition performance.
Contribution
It presents a novel few-shot learning approach that transfers knowledge from a large-scale pre-trained model to enable personalized gesture addition on wearable devices.
Findings
Achieved 95.7% accuracy in large-scale gesture recognition
Demonstrated effective on-device customization with 1-5 shots
Validated usability and learnability through user studies
Abstract
We present a framework for gesture customization requiring minimal examples from users, all without degrading the performance of existing gesture sets. To achieve this, we first deployed a large-scale study (N=500+) to collect data and train an accelerometer-gyroscope recognition model with a cross-user accuracy of 95.7% and a false-positive rate of 0.6 per hour when tested on everyday non-gesture data. Next, we design a few-shot learning framework which derives a lightweight model from our pre-trained model, enabling knowledge transfer without performance degradation. We validate our approach through a user study (N=20) examining on-device customization from 12 new gestures, resulting in an average accuracy of 55.3%, 83.1%, and 87.2% on using one, three, or five shots when adding a new gesture, while maintaining the same recognition accuracy and false-positive rate from the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
