Mask-off: Synthesizing Face Images in the Presence of Head-mounted Displays

TL;DR

This paper introduces a novel hardware setup and synthesis framework to generate realistic, occlusion-free face images in augmented reality scenarios involving head-mounted displays, enhancing user experience in telecommunication and gaming.

Contribution

The paper presents a new multi-camera setup and a robust synthesis framework with innovative algorithms for 3D head reconstruction, face alignment, and eye colorization under occlusion.

Findings

Hardware setup effectively captures face and eye regions

Synthesized images are realistic and robust in video sequences

Algorithms accurately reproduce eye movements and wrinkles

Abstract

A head-mounted display (HMD) could be an important component of augmented reality system. However, as the upper face region is seriously occluded by the device, the user experience could be affected in applications such as telecommunication and multi-player video games. In this paper, we first present a novel experimental setup that consists of two near-infrared (NIR) cameras to point to the eye regions and one visible-light RGB camera to capture the visible face region. The main purpose of this paper is to synthesize realistic face images without occlusions based on the images captured by these cameras. To this end, we propose a novel synthesis framework that contains four modules: 3D head reconstruction, face alignment and tracking, face synthesis, and eye synthesis. In face synthesis, we propose a novel algorithm that can robustly align and track a personalized 3D head model given a face that is severely occluded by the HMD. In eye synthesis, in order to generate accurate eye movements and dynamic wrinkle variations around eye regions, we propose another novel algorithm to colorize the NIR eye images and further remove the "red eye" effects caused by the colorization. Results show that both hardware setup and system framework are robust to synthesize realistic face images in video sequences.