TapType: Ten-finger text entry on everyday surfaces via Bayesian inference

TL;DR

TapType enables full-size ten-finger text entry on passive surfaces using wrist sensors and Bayesian inference, achieving competitive typing speeds and low error rates without a physical keyboard.

Contribution

This paper introduces TapType, a novel mobile text entry system that decodes finger taps on passive surfaces into text using Bayesian neural networks and language models.

Findings

Participants typed 19 WPM on average after training.

Expert typists exceeded 25 WPM with low error rates.

System demonstrated effective use in mobile, MR, and eyes-free contexts.

Abstract

Despite the advent of touchscreens, typing on physical keyboards remains most efficient for entering text, because users can leverage all fingers across a full-size keyboard for convenient typing. As users increasingly type on the go, text input on mobile and wearable devices has had to compromise on full-size typing. In this paper, we present TapType, a mobile text entry system for full-size typing on passive surfaces--without an actual keyboard. From the inertial sensors inside a band on either wrist, TapType decodes and relates surface taps to a traditional QWERTY keyboard layout. The key novelty of our method is to predict the most likely character sequences by fusing the finger probabilities from our Bayesian neural network classifier with the characters' prior probabilities from an n-gram language model. In our online evaluation, participants on average typed 19 words per minute with a character error rate of 0.6% after 30 minutes of training. Expert typists thereby consistently achieved more than 25 WPM at a similar error rate. We demonstrate applications of TapType in mobile use around smartphones and tablets, as a complement to interaction in situated Mixed Reality outside visual control, and as an eyes-free mobile text input method using an audio feedback-only interface.