Non-Deterministic Face Mask Removal Based On 3D Priors

TL;DR

This paper introduces a novel face mask removal method that leverages 3D face priors to produce high-quality, non-deterministic inpainting results, overcoming limitations of manual region labeling and deterministic outputs.

Contribution

It integrates 3D face reconstruction with inpainting to enable dynamic, multi-expression face mask removal without manual region annotation.

Findings

Effective inpainting of masked faces with diverse expressions.

Outperforms existing methods in qualitative and quantitative evaluations.

Produces non-deterministic results for varied face reconstructions.

Abstract

This paper presents a novel image inpainting framework for face mask removal. Although current methods have demonstrated their impressive ability in recovering damaged face images, they suffer from two main problems: the dependence on manually labeled missing regions and the deterministic result corresponding to each input. The proposed approach tackles these problems by integrating a multi-task 3D face reconstruction module with a face inpainting module. Given a masked face image, the former predicts a 3DMM-based reconstructed face together with a binary occlusion map, providing dense geometrical and textural priors that greatly facilitate the inpainting task of the latter. By gradually controlling the 3D shape parameters, our method generates high-quality dynamic inpainting results with different expressions and mouth movements. Qualitative and quantitative experiments verify the effectiveness of the proposed method.