AUGlasses: Continuous Action Unit based Facial Reconstruction with Low-power IMUs on Smart Glasses

TL;DR

AUGlasses introduces a low-power, unobtrusive system using IMUs and deep learning to accurately reconstruct facial expressions in real-time for augmented reality applications.

Contribution

The paper presents a novel low-power IMU-based approach combined with transformer models for continuous facial reconstruction on smart glasses.

Findings

Accurately predicts 14 AU intensities with MAE of 0.187

Achieves facial reconstruction with MAE of 1.93 mm

Demonstrates robust performance across users

Abstract

Recent advancements in augmented reality (AR) have enabled the use of various sensors on smart glasses for applications like facial reconstruction, which is vital to improve AR experiences for virtual social activities. However, the size and power constraints of smart glasses demand a miniature and low-power sensing solution. AUGlasses achieves unobtrusive low-power facial reconstruction by placing inertial measurement units (IMU) against the temporal area on the face to capture the skin deformations, which are caused by facial muscle movements. These IMU signals, along with historical data on facial action units (AUs), are processed by a transformer-based deep learning model to estimate AU intensities in real-time, which are then used for facial reconstruction. Our results show that AUGlasses accurately predicts the strength (0-5 scale) of 14 key AUs with a cross-user mean absolute error (MAE) of 0.187 (STD = 0.025) and achieves facial reconstruction with a cross-user MAE of 1.93 mm (STD = 0.353). We also integrated various preprocessing and training techniques to ensure robust performance for continuous sensing. Micro-benchmark tests indicate that our system consistently performs accurate continuous facial reconstruction with a fine-tuned cross-user model, achieving an AU MAE of 0.35.