Augmenting Mobile Phone Interaction with Face-Engaged Gestures

TL;DR

This paper explores face-engaged gestures detected via smartphone front cameras, combining head movements with touch and motion inputs to expand mobile interaction possibilities, including touch-free and one-handed use.

Contribution

Introduces novel face-engaged interaction techniques that augment traditional mobile inputs by leveraging head movements with touch and device sensors.

Findings

Users reacted positively to the new interaction techniques.

The techniques enable touch-free and one-handed interactions.

The methods expand the mobile interaction vocabulary.

Abstract

The movement of a user's face, easily detected by a smartphone's front camera, is an underexploited input modality for mobile interactions. We introduce three sets of face-engaged interaction techniques for augmenting the traditional mobile inputs, which leverages the combination of the head movements with touch gestures and device motions, all sensed via the phone's built-in sensors. We systematically present the space of design considerations for mobile interactions using one or more of the three input modalities (i.e., touch, motion, and head). The additional affordances of the proposed techniques expand the mobile interaction vocabulary, and can facilitate unique usage scenarios such as one-hand or touch-free interaction. An initial evaluation was conducted and users had positive reactions to the new techniques, indicating the promise of an intuitive and convenient user experience.