TL;DR
This paper presents a low-cost, real-time system for reconstructing and displaying facial expressions in VR telepresence using open source software and affordable hardware, enhancing face-to-face communication in VR.
Contribution
It introduces one of the first low-cost, end-to-end face reconstruction systems for VR that uses CNNs and GANs with commodity hardware, making high-fidelity face avatars more accessible.
Findings
Capable of real-time face expression generation on standard gaming PCs.
Achieves personalized facial avatars with distinct movements and expressions.
Lower fidelity compared to high-end research prototypes, but significantly more affordable.
Abstract
Face-to-face conversation in Virtual Reality (VR) is a challenge when participants wear head-mounted displays (HMD). A significant portion of a participant's face is hidden and facial expressions are difficult to perceive. Past research has shown that high-fidelity face reconstruction with personal avatars in VR is possible under laboratory conditions with high-cost hardware. In this paper, we propose one of the first low-cost systems for this task which uses only open source, free software and affordable hardware. Our approach is to track the user's face underneath the HMD utilizing a Convolutional Neural Network (CNN) and generate corresponding expressions with Generative Adversarial Networks (GAN) for producing RGBD images of the person's face. We use commodity hardware with low-cost extensions such as 3D-printed mounts and miniature cameras. Our approach learns end-to-end without…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
