Gradient Adjusting Networks for Domain Inversion

TL;DR

This paper introduces a novel per-image optimization method that fine-tunes StyleGAN2 generators using shallow update networks, achieving near-perfect inversion of real images while preserving editing capabilities, outperforming existing methods.

Contribution

The paper proposes Gradient Modification Modules for efficient generator tuning, enabling high-quality image inversion without sacrificing editing functionality.

Findings

Significant improvement over state-of-the-art inversion methods.

Efficient one-shot training of shallow update networks.

Maintains editing capabilities after generator modification.

Abstract

StyleGAN2 was demonstrated to be a powerful image generation engine that supports semantic editing. However, in order to manipulate a real-world image, one first needs to be able to retrieve its corresponding latent representation in StyleGAN's latent space that is decoded to an image as close as possible to the desired image. For many real-world images, a latent representation does not exist, which necessitates the tuning of the generator network. We present a per-image optimization method that tunes a StyleGAN2 generator such that it achieves a local edit to the generator's weights, resulting in almost perfect inversion, while still allowing image editing, by keeping the rest of the mapping between an input latent representation tensor and an output image relatively intact. The method is based on a one-shot training of a set of shallow update networks (aka. Gradient Modification Modules) that modify the layers of the generator. After training the Gradient Modification Modules, a modified generator is obtained by a single application of these networks to the original parameters, and the previous editing capabilities of the generator are maintained. Our experiments show a sizable gap in performance over the current state of the art in this very active domain. Our code is available at \url{https://github.com/sheffier/gani}.