ExpresSense: Exploring a Standalone Smartphone to Sense Engagement of Users from Facial Expressions Using Acoustic Sensing

TL;DR

ExpresSense is a lightweight smartphone application that uses acoustic signals to detect facial expressions with about 75% accuracy, addressing limitations of image-based methods like occlusion and privacy concerns.

Contribution

This work introduces a novel acoustic sensing approach for facial expression detection on standalone smartphones, bypassing traditional image processing limitations.

Findings

Achieved ~75% classification accuracy on various expressions.

Validated effectiveness through lab and large-scale studies.

Demonstrated potential for privacy-preserving expression sensing.

Abstract

Facial expressions have been considered a metric reflecting a person's engagement with a task. While the evolution of expression detection methods is consequential, the foundation remains mostly on image processing techniques that suffer from occlusion, ambient light, and privacy concerns. In this paper, we propose ExpresSense, a lightweight application for standalone smartphones that relies on near-ultrasound acoustic signals for detecting users' facial expressions. ExpresSense has been tested on different users in lab-scaled and large-scale studies for both posed as well as natural expressions. By achieving a classification accuracy of ~75% over various basic expressions, we discuss the potential of a standalone smartphone to sense expressions through acoustic sensing.