High Resolution Face Completion with Multiple Controllable Attributes via Fully End-to-End Progressive Generative Adversarial Networks

TL;DR

This paper introduces a fully end-to-end progressive GAN framework for high-resolution face completion with multiple controllable attributes, achieving sharp, realistic results efficiently and outperforming existing methods.

Contribution

The work proposes a novel progressive training strategy, new network architectures, and loss functions for controllable, high-resolution face completion with improved quality and speed.

Findings

Achieves 0.007 seconds inference time at 1024x1024 resolution.

Outperforms state-of-the-art methods in human studies.

Effectively handles large structural and appearance variations.

Abstract

We present a deep learning approach for high resolution face completion with multiple controllable attributes (e.g., male and smiling) under arbitrary masks. Face completion entails understanding both structural meaningfulness and appearance consistency locally and globally to fill in "holes" whose content do not appear elsewhere in an input image. It is a challenging task with the difficulty level increasing significantly with respect to high resolution, the complexity of "holes" and the controllable attributes of filled-in fragments. Our system addresses the challenges by learning a fully end-to-end framework that trains generative adversarial networks (GANs) progressively from low resolution to high resolution with conditional vectors encoding controllable attributes. We design novel network architectures to exploit information across multiple scales effectively and efficiently. We introduce new loss functions encouraging sharp completion. We show that our system can complete faces with large structural and appearance variations using a single feed-forward pass of computation with mean inference time of 0.007 seconds for images at 1024 x 1024 resolution. We also perform a pilot human study that shows our approach outperforms state-of-the-art face completion methods in terms of rank analysis. The code will be released upon publication.