Diverse Inpainting and Editing with GAN Inversion

TL;DR

This paper introduces a novel method for inverting erased images into StyleGAN's latent space, enabling realistic and diverse inpainting and editing by combining features from erased images and random samples.

Contribution

The paper proposes a new encoder and mixing network architecture trained with a novel setup to improve inpainting diversity and realism for erased images in StyleGAN.

Findings

Significant improvement in inpainting quality over state-of-the-art methods

Enhanced diversity in generated inpaintings through latent code augmentation

Better color consistency between inpainted and original regions

Abstract

Recent inversion methods have shown that real images can be inverted into StyleGAN's latent space and numerous edits can be achieved on those images thanks to the semantically rich feature representations of well-trained GAN models. However, extensive research has also shown that image inversion is challenging due to the trade-off between high-fidelity reconstruction and editability. In this paper, we tackle an even more difficult task, inverting erased images into GAN's latent space for realistic inpaintings and editings. Furthermore, by augmenting inverted latent codes with different latent samples, we achieve diverse inpaintings. Specifically, we propose to learn an encoder and mixing network to combine encoded features from erased images with StyleGAN's mapped features from random samples. To encourage the mixing network to utilize both inputs, we train the networks with generated data via a novel set-up. We also utilize higher-rate features to prevent color inconsistencies between the inpainted and unerased parts. We run extensive experiments and compare our method with state-of-the-art inversion and inpainting methods. Qualitative metrics and visual comparisons show significant improvements.