Sketch Your Own GAN

TL;DR

This paper introduces GAN Sketching, a method allowing users to modify pre-trained GANs using sketches, enabling easy customization of generated images without extensive datasets or deep learning expertise.

Contribution

The paper presents a novel approach to rewrite GANs with sketches, making GAN customization accessible to novices by adjusting model weights based on user sketches.

Findings

Successfully molds GANs to match sketch shapes and poses

Maintains realism and diversity in generated images

Enables applications like latent space interpolation and image editing

Abstract

Can a user create a deep generative model by sketching a single example? Traditionally, creating a GAN model has required the collection of a large-scale dataset of exemplars and specialized knowledge in deep learning. In contrast, sketching is possibly the most universally accessible way to convey a visual concept. In this work, we present a method, GAN Sketching, for rewriting GANs with one or more sketches, to make GANs training easier for novice users. In particular, we change the weights of an original GAN model according to user sketches. We encourage the model's output to match the user sketches through a cross-domain adversarial loss. Furthermore, we explore different regularization methods to preserve the original model's diversity and image quality. Experiments have shown that our method can mold GANs to match shapes and poses specified by sketches while maintaining realism and diversity. Finally, we demonstrate a few applications of the resulting GAN, including latent space interpolation and image editing.