TapNet: The Design, Training, Implementation, and Applications of a Multi-Task Learning CNN for Off-Screen Mobile Input

TL;DR

TapNet is a multi-task deep learning model that uses built-in mobile sensors to accurately detect and classify off-screen taps, enabling practical one-handed mobile interactions without additional hardware.

Contribution

We introduce TapNet, a novel multi-task CNN that jointly detects tap events and recognizes tap properties using only built-in IMU sensors on smartphones.

Findings

TapNet significantly outperforms existing methods in tap detection accuracy.

The model effectively recognizes tap direction and location simultaneously.

Our datasets and experiments establish a new foundation for off-screen mobile input research.

Abstract

To make off-screen interaction without specialized hardware practical, we investigate using deep learning methods to process the common built-in IMU sensor (accelerometers and gyroscopes) on mobile phones into a useful set of one-handed interaction events. We present the design, training, implementation and applications of TapNet, a multi-task network that detects tapping on the smartphone. With phone form factor as auxiliary information, TapNet can jointly learn from data across devices and simultaneously recognize multiple tap properties, including tap direction and tap location. We developed two datasets consisting of over 135K training samples, 38K testing samples, and 32 participants in total. Experimental evaluation demonstrated the effectiveness of the TapNet design and its significant improvement over the state of the art. Along with the datasets, (https://sites.google.com/site/michaelxlhuang/datasets/tapnet-dataset), and extensive experiments, TapNet establishes a new technical foundation for off-screen mobile input.