SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis

TL;DR

SketchyGAN introduces a novel GAN-based method for generating diverse, realistic images from sketches across multiple categories, utilizing automatic data augmentation and a new network block to enhance image quality.

Contribution

The paper presents a new GAN architecture with a multi-scale input injection block and an automatic sketch data augmentation technique for improved sketch-to-image synthesis.

Findings

Produces more realistic images than previous methods

Achieves higher Inception Scores on multiple categories

Effective data augmentation improves synthesis quality

Abstract

Synthesizing realistic images from human drawn sketches is a challenging problem in computer graphics and vision. Existing approaches either need exact edge maps, or rely on retrieval of existing photographs. In this work, we propose a novel Generative Adversarial Network (GAN) approach that synthesizes plausible images from 50 categories including motorcycles, horses and couches. We demonstrate a data augmentation technique for sketches which is fully automatic, and we show that the augmented data is helpful to our task. We introduce a new network building block suitable for both the generator and discriminator which improves the information flow by injecting the input image at multiple scales. Compared to state-of-the-art image translation methods, our approach generates more realistic images and achieves significantly higher Inception Scores.