Unsupervised Sketch-to-Photo Synthesis

TL;DR

This paper introduces an unsupervised method for converting free-hand sketches into realistic photos by decomposing the task into shape translation and content enrichment, leveraging self-supervision and attention mechanisms.

Contribution

It presents the first unsupervised sketch-to-photo synthesis approach, addressing the challenge of unpaired data and sketch abstraction through a two-stage translation framework.

Findings

Achieves sketch-faithful and photo-realistic synthesis.

Enables sketch-based image retrieval.

Produces a universal sketch generator capturing human perception.

Abstract

Humans can envision a realistic photo given a free-hand sketch that is not only spatially imprecise and geometrically distorted but also without colors and visual details. We study unsupervised sketch-to-photo synthesis for the first time, learning from unpaired sketch-photo data where the target photo for a sketch is unknown during training. Existing works only deal with style change or spatial deformation alone, synthesizing photos from edge-aligned line drawings or transforming shapes within the same modality, e.g., color images. Our key insight is to decompose unsupervised sketch-to-photo synthesis into a two-stage translation task: First shape translation from sketches to grayscale photos and then content enrichment from grayscale to color photos. We also incorporate a self-supervised denoising objective and an attention module to handle abstraction and style variations that are inherent and specific to sketches. Our synthesis is sketch-faithful and photo-realistic to enable sketch-based image retrieval in practice. An exciting corollary product is a universal and promising sketch generator that captures human visual perception beyond the edge map of a photo.