Facial De-occlusion Network for Virtual Telepresence Systems

TL;DR

This paper introduces a face de-occlusion network designed for virtual telepresence, enabling real-time, photo-realistic face reconstruction in VR environments where headsets occlude facial features.

Contribution

The paper presents a novel deep learning-based method specifically addressing face occlusion in VR, improving upon existing inpainting techniques for real-time applications.

Findings

Achieves real-time de-occlusion of facial features in VR.

Produces photo-realistic face reconstructions despite occlusions.

Outperforms previous inpainting methods for face de-occlusion.

Abstract

To see what is not in the image is one of the broader missions of computer vision. Technology to inpaint images has made significant progress with the coming of deep learning. This paper proposes a method to tackle occlusion specific to human faces. Virtual presence is a promising direction in communication and recreation for the future. However, Virtual Reality (VR) headsets occlude a significant portion of the face, hindering the photo-realistic appearance of the face in the virtual world. State-of-the-art image inpainting methods for de-occluding the eye region does not give usable results. To this end, we propose a working solution that gives usable results to tackle this problem enabling the use of the real-time photo-realistic de-occluded face of the user in VR settings.