Semantics-Guided Object Removal for Facial Images: with Broad Applicability and Robust Style Preservation

TL;DR

This paper introduces SGIN, a semantics-guided inpainting network that effectively removes objects from facial images, maintaining style consistency and fine details across various occlusion sizes.

Contribution

The paper proposes a novel model combining the strengths of U-net and modulated generators, guided by semantic maps, for improved object removal in facial images.

Findings

Handles occlusions of any size while preserving style and details.

Outperforms existing methods in style consistency and detail preservation.

Provides semantic control for facial feature manipulation.

Abstract

Object removal and image inpainting in facial images is a task in which objects that occlude a facial image are specifically targeted, removed, and replaced by a properly reconstructed facial image. Two different approaches utilizing U-net and modulated generator respectively have been widely endorsed for this task for their unique advantages but notwithstanding each method's innate disadvantages. U-net, a conventional approach for conditional GANs, retains fine details of unmasked regions but the style of the reconstructed image is inconsistent with the rest of the original image and only works robustly when the size of the occluding object is small enough. In contrast, the modulated generative approach can deal with a larger occluded area in an image and provides {a} more consistent style, yet it usually misses out on most of the detailed features. This trade-off between these two models necessitates an invention of a model that can be applied to any size of mask while maintaining a consistent style and preserving minute details of facial features. Here, we propose Semantics-Guided Inpainting Network (SGIN) which itself is a modification of the modulated generator, aiming to take advantage of its advanced generative capability and preserve the high-fidelity details of the original image. By using the guidance of a semantic map, our model is capable of manipulating facial features which grants direction to the one-to-many problem for further practicability.