Can Capacitive Touch Images Enhance Mobile Keyboard Decoding?

TL;DR

This study explores whether using full touch heatmaps, instead of just touch centroids, can improve the accuracy and user experience of mobile keyboard decoding, showing significant performance gains.

Contribution

The paper demonstrates that incorporating touch heatmaps into machine learning models enhances decoding accuracy and user satisfaction in mobile keyboards.

Findings

21.4% reduction in character error rate with heatmaps

Lower error rate and faster typing speed in live user study

Higher user satisfaction with heatmap-based decoding

Abstract

Capacitive touch sensors capture the two-dimensional spatial profile (referred to as a touch heatmap) of a finger's contact with a mobile touchscreen. However, the research and design of touchscreen mobile keyboards -- one of the most speed and accuracy demanding touch interfaces -- has focused on the location of the touch centroid derived from the touch image heatmap as the input, discarding the rest of the raw spatial signals. In this paper, we investigate whether touch heatmaps can be leveraged to further improve the tap decoding accuracy for mobile touchscreen keyboards. Specifically, we developed and evaluated machine-learning models that interpret user taps by using the centroids and/or the heatmaps as their input and studied the contribution of the heatmaps to model performance. The results show that adding the heatmap into the input feature set led to 21.4% relative reduction of character error rates on average, compared to using the centroid alone. Furthermore, we conducted a live user study with the centroid-based and heatmap-based decoders built into Pixel 6 Pro devices and observed lower error rate, faster typing speed, and higher self-reported satisfaction score based on the heatmap-based decoder than the centroid-based decoder. These findings underline the promise of utilizing touch heatmaps for improving typing experience in mobile keyboards.