IMUFace: Towards Always-On 3D Facial Reconstruction via Earphone Inertial Sensing

TL;DR

IMUFace leverages earphone-embedded inertial sensors and deep learning to enable covert, low-power 3D facial reconstruction, demonstrating high accuracy with minimal training data and real-world applicability.

Contribution

This work introduces IMUFace, a novel system using inertial sensors in earphones for facial reconstruction, improving aesthetics and power efficiency over camera-based methods.

Findings

Achieves 2.21 mm landmark prediction accuracy

Operates at 30 Hz sampling rate with 58 mW power consumption

Requires only five minutes of training data

Abstract

The potential of facial expression reconstruction technology is significant, with applications in various fields such as human-computer interaction, affective computing, and virtual reality. Recent studies have proposed using ear-worn devices for facial expression reconstruction to address the environmental limitations and privacy concerns associated with traditional camera-based methods. However, these approaches still require improvements in terms of aesthetics and power consumption. This paper introduces a system called IMUFace. It uses inertial measurement units (IMUs) embedded in wireless earphones to detect subtle ear movements caused by facial muscle activities, allowing for covert and low-power facial reconstruction. A user study involving 12 participants was conducted, and a deep learning model named IMUTwinTrans was proposed. The results show that IMUFace can accurately predict users' facial landmarks with a precision of 2.21 mm, using only five minutes of training data. The predicted landmarks can be utilized to reconstruct a three-dimensional facial model. IMUFace operates at a sampling rate of 30 Hz with a relatively low power consumption of 58 mW. The findings presented in this study demonstrate the real-world applicability of IMUFace and highlight potential directions for further research to facilitate its practical adoption.