Picture that Sketch: Photorealistic Image Generation from Abstract Sketches

TL;DR

This paper introduces a novel method for converting abstract, free-hand sketches into photorealistic images using a decoupled encoder-decoder framework with StyleGAN, making sketch-to-photo generation accessible to amateurs.

Contribution

It presents a new decoupled training paradigm and an autoregressive sketch mapper that effectively bridge the abstraction gap in sketch-to-photo synthesis, enabling photorealistic outputs from simple sketches.

Findings

Achieved photorealistic image generation from amateur sketches.

Surpassed state-of-the-art in sketch-based image retrieval tasks.

Democratized sketch-to-photo pipeline for non-expert users.

Abstract

Given an abstract, deformed, ordinary sketch from untrained amateurs like you and me, this paper turns it into a photorealistic image - just like those shown in Fig. 1(a), all non-cherry-picked. We differ significantly from prior art in that we do not dictate an edgemap-like sketch to start with, but aim to work with abstract free-hand human sketches. In doing so, we essentially democratise the sketch-to-photo pipeline, "picturing" a sketch regardless of how good you sketch. Our contribution at the outset is a decoupled encoder-decoder training paradigm, where the decoder is a StyleGAN trained on photos only. This importantly ensures that generated results are always photorealistic. The rest is then all centred around how best to deal with the abstraction gap between sketch and photo. For that, we propose an autoregressive sketch mapper trained on sketch-photo pairs that maps a sketch to the StyleGAN latent space. We further introduce specific designs to tackle the abstract nature of human sketches, including a fine-grained discriminative loss on the back of a trained sketch-photo retrieval model, and a partial-aware sketch augmentation strategy. Finally, we showcase a few downstream tasks our generation model enables, amongst them is showing how fine-grained sketch-based image retrieval, a well-studied problem in the sketch community, can be reduced to an image (generated) to image retrieval task, surpassing state-of-the-arts. We put forward generated results in the supplementary for everyone to scrutinise.