AirWare: Utilizing Embedded Audio and Infrared Signals for In-Air Hand-Gesture Recognition

TL;DR

AirWare is a gesture recognition system that uses embedded audio and infrared sensors in devices to classify in-air hand gestures with neural networks, aiming for a sensor-free solution but facing reliability challenges.

Contribution

This work introduces AirWare, a novel in-air gesture recognition system leveraging existing device sensors and CNNs, eliminating the need for external sensors.

Findings

AirWare achieves over 80% accuracy on smaller gesture sets.

The system struggles with classifying all 21 gestures reliably.

Performance improves when focusing on subsets of gestures.

Abstract

We introduce AirWare, an in-air hand-gesture recognition system that uses the already embedded speaker and microphone in most electronic devices, together with embedded infrared proximity sensors. Gestures identified by AirWare are performed in the air above a touchscreen or a mobile phone. AirWare utilizes convolutional neural networks to classify a large vocabulary of hand gestures using multi-modal audio Doppler signatures and infrared (IR) sensor information. As opposed to other systems which use high frequency Doppler radars or depth cameras to uniquely identify in-air gestures, AirWare does not require any external sensors. In our analysis, we use openly available APIs to interface with the Samsung Galaxy S5 audio and proximity sensors for data collection. We find that AirWare is not reliable enough for a deployable interaction system when trying to classify a gesture set of 21 gestures, with an average true positive rate of only 50.5% per gesture. To improve performance, we train AirWare to identify subsets of the 21 gestures vocabulary based on possible usage scenarios. We find that AirWare can identify three gesture sets with average true positive rate greater than 80% using 4--7 gestures per set, which comprises a vocabulary of 16 unique in-air gestures.