FacialMotionID: Identifying Users of Mixed Reality Headsets using Abstract Facial Motion Representations

TL;DR

This study demonstrates that abstract facial motion data from mixed reality headsets can reliably identify users and infer emotional states, highlighting significant privacy risks in immersive virtual environments.

Contribution

It provides empirical evidence that facial motion representations in mixed reality can be used for user identification and emotion inference, raising privacy concerns.

Findings

Up to 98% user re-identification accuracy

Able to identify users across different device types

Emotional states inferred with up to 86% accuracy

Abstract

Facial motion capture in mixed reality headsets enables real-time avatar animation, allowing users to convey non-verbal cues during virtual interactions. However, as facial motion data constitutes a behavioral biometric, its use raises novel privacy concerns. With mixed reality systems becoming more immersive and widespread, understanding whether face motion data can lead to user identification or inference of sensitive attributes is increasingly important. To address this, we conducted a study with 116 participants using three types of headsets across three sessions, collecting facial, eye, and head motion data during verbal and non-verbal tasks. The data used is not raw video, but rather, abstract representations that are used to animate digital avatars. Our analysis shows that individuals can be re-identified from this data with up to 98% balanced accuracy, are even identifiable across device types, and that emotional states can be inferred with up to 86% accuracy. These results underscore the potential privacy risks inherent in face motion tracking in mixed reality environments.