Eye In-Painting with Exemplar Generative Adversarial Networks

TL;DR

This paper presents Exemplar GANs (ExGANs), a novel conditional GAN framework that uses exemplar information to produce high-quality, personalized in-painting results, demonstrated on eye in-painting with a new benchmark dataset.

Contribution

Introduction of ExGANs, a new conditional GAN architecture that incorporates exemplar information at multiple points for improved personalized in-painting.

Findings

ExGANs generate photo-realistic, personalized in-painting results.

ExGANs outperform previous methods in eye in-painting tasks.

A new benchmark dataset for eye in-painting is introduced.

Abstract

This paper introduces a novel approach to in-painting where the identity of the object to remove or change is preserved and accounted for at inference time: Exemplar GANs (ExGANs). ExGANs are a type of conditional GAN that utilize exemplar information to produce high-quality, personalized in painting results. We propose using exemplar information in the form of a reference image of the region to in-paint, or a perceptual code describing that object. Unlike previous conditional GAN formulations, this extra information can be inserted at multiple points within the adversarial network, thus increasing its descriptive power. We show that ExGANs can produce photo-realistic personalized in-painting results that are both perceptually and semantically plausible by applying them to the task of closed to-open eye in-painting in natural pictures. A new benchmark dataset is also introduced for the task of eye in-painting for future comparisons.