Rewriting Geometric Rules of a GAN

TL;DR

This paper introduces a method for user-guided geometric editing of GANs by applying low-rank updates and style-mixing, enabling the creation of customized generative models with minimal data and interactive composition.

Contribution

It proposes a novel low-rank update technique for GAN editing, combined with style-mixing for overfitting prevention, allowing flexible, user-driven model customization without large datasets.

Findings

Outperforms recent GAN fine-tuning methods in empirical tests.

Enables creation of models with specific geometric transformations.

Supports model composition for complex effects.

Abstract

Deep generative models make visual content creation more accessible to novice users by automating the synthesis of diverse, realistic content based on a collected dataset. However, the current machine learning approaches miss a key element of the creative process -- the ability to synthesize things that go far beyond the data distribution and everyday experience. To begin to address this issue, we enable a user to "warp" a given model by editing just a handful of original model outputs with desired geometric changes. Our method applies a low-rank update to a single model layer to reconstruct edited examples. Furthermore, to combat overfitting, we propose a latent space augmentation method based on style-mixing. Our method allows a user to create a model that synthesizes endless objects with defined geometric changes, enabling the creation of a new generative model without the burden of curating a large-scale dataset. We also demonstrate that edited models can be composed to achieve aggregated effects, and we present an interactive interface to enable users to create new models through composition. Empirical measurements on multiple test cases suggest the advantage of our method against recent GAN fine-tuning methods. Finally, we showcase several applications using the edited models, including latent space interpolation and image editing.