Designing an Encoder for StyleGAN Image Manipulation

TL;DR

This paper studies the StyleGAN latent space to improve real image inversion for editing, balancing tradeoffs between distortion and perceptual quality, resulting in better editing performance with minimal reconstruction loss.

Contribution

It introduces principles for encoder design that control inversion proximity to training regions, enhancing real image editing quality with a novel encoder tailored for StyleGAN.

Findings

Improved real image inversion quality on challenging domains.

Balanced tradeoffs enable effective editing with minimal reconstruction loss.

Proposed encoder outperforms existing methods in qualitative and quantitative evaluations.

Abstract

Recently, there has been a surge of diverse methods for performing image editing by employing pre-trained unconditional generators. Applying these methods on real images, however, remains a challenge, as it necessarily requires the inversion of the images into their latent space. To successfully invert a real image, one needs to find a latent code that reconstructs the input image accurately, and more importantly, allows for its meaningful manipulation. In this paper, we carefully study the latent space of StyleGAN, the state-of-the-art unconditional generator. We identify and analyze the existence of a distortion-editability tradeoff and a distortion-perception tradeoff within the StyleGAN latent space. We then suggest two principles for designing encoders in a manner that allows one to control the proximity of the inversions to regions that StyleGAN was originally trained on. We present an encoder based on our two principles that is specifically designed for facilitating editing on real images by balancing these tradeoffs. By evaluating its performance qualitatively and quantitatively on numerous challenging domains, including cars and horses, we show that our inversion method, followed by common editing techniques, achieves superior real-image editing quality, with only a small reconstruction accuracy drop.