Unmasking Communication Partners: A Low-Cost AI Solution for Digitally Removing Head-Mounted Displays in VR-Based Telepresence

TL;DR

This paper presents a low-cost, real-time system for reconstructing and displaying facial expressions in VR telepresence using open source software and affordable hardware, enhancing face-to-face communication in VR.

Contribution

It introduces one of the first low-cost, end-to-end face reconstruction systems for VR that uses CNNs and GANs with commodity hardware, making high-fidelity face avatars more accessible.

Findings

Capable of real-time face expression generation on standard gaming PCs.

Achieves personalized facial avatars with distinct movements and expressions.

Lower fidelity compared to high-end research prototypes, but significantly more affordable.

Abstract

Face-to-face conversation in Virtual Reality (VR) is a challenge when participants wear head-mounted displays (HMD). A significant portion of a participant's face is hidden and facial expressions are difficult to perceive. Past research has shown that high-fidelity face reconstruction with personal avatars in VR is possible under laboratory conditions with high-cost hardware. In this paper, we propose one of the first low-cost systems for this task which uses only open source, free software and affordable hardware. Our approach is to track the user's face underneath the HMD utilizing a Convolutional Neural Network (CNN) and generate corresponding expressions with Generative Adversarial Networks (GAN) for producing RGBD images of the person's face. We use commodity hardware with low-cost extensions such as 3D-printed mounts and miniature cameras. Our approach learns end-to-end without manual intervention, runs in real time, and can be trained and executed on an ordinary gaming computer. We report evaluation results showing that our low-cost system does not achieve the same fidelity of research prototypes using high-end hardware and closed source software, but it is capable of creating individual facial avatars with person-specific characteristics in movements and expressions.