MaskGAN: Towards Diverse and Interactive Facial Image Manipulation

TL;DR

MaskGAN introduces a flexible, interactive facial image manipulation framework using semantic masks, enabling diverse results and robustness to user edits, validated on a new high-resolution face dataset.

Contribution

The paper presents MaskGAN, a novel framework with Dense Mapping Network and Editing Behavior Simulated Training, for diverse and interactive face manipulation using semantic masks.

Findings

Outperforms state-of-the-art methods on attribute transfer and style copy tasks.

Introduces CelebAMask-HQ, a large-scale high-resolution face dataset with detailed mask annotations.

Demonstrates robustness to various user manipulations through dual-editing consistency.

Abstract

Facial image manipulation has achieved great progress in recent years. However, previous methods either operate on a predefined set of face attributes or leave users little freedom to interactively manipulate images. To overcome these drawbacks, we propose a novel framework termed MaskGAN, enabling diverse and interactive face manipulation. Our key insight is that semantic masks serve as a suitable intermediate representation for flexible face manipulation with fidelity preservation. MaskGAN has two main components: 1) Dense Mapping Network (DMN) and 2) Editing Behavior Simulated Training (EBST). Specifically, DMN learns style mapping between a free-form user modified mask and a target image, enabling diverse generation results. EBST models the user editing behavior on the source mask, making the overall framework more robust to various manipulated inputs. Specifically, it introduces dual-editing consistency as the auxiliary supervision signal. To facilitate extensive studies, we construct a large-scale high-resolution face dataset with fine-grained mask annotations named CelebAMask-HQ. MaskGAN is comprehensively evaluated on two challenging tasks: attribute transfer and style copy, demonstrating superior performance over other state-of-the-art methods. The code, models, and dataset are available at https://github.com/switchablenorms/CelebAMask-HQ.