Unsupervised Scene Sketch to Photo Synthesis

TL;DR

This paper introduces an unsupervised method to generate realistic photos from scene sketches by standardizing data and disentangling structure and style, enabling high-fidelity photo synthesis and controllable editing.

Contribution

The method uniquely leverages a standardization module to create pseudo pairs and disentangles structure and style for flexible, high-quality photo synthesis from sketches without paired data.

Findings

Outperforms state-of-the-art photo synthesis methods in perceptual metrics.

Enables fine-grained, stroke-level editing of synthesized photos.

Produces realistic images with high fidelity from scene sketches.

Abstract

Sketches make an intuitive and powerful visual expression as they are fast executed freehand drawings. We present a method for synthesizing realistic photos from scene sketches. Without the need for sketch and photo pairs, our framework directly learns from readily available large-scale photo datasets in an unsupervised manner. To this end, we introduce a standardization module that provides pseudo sketch-photo pairs during training by converting photos and sketches to a standardized domain, i.e. the edge map. The reduced domain gap between sketch and photo also allows us to disentangle them into two components: holistic scene structures and low-level visual styles such as color and texture. Taking this advantage, we synthesize a photo-realistic image by combining the structure of a sketch and the visual style of a reference photo. Extensive experimental results on perceptual similarity metrics and human perceptual studies show the proposed method could generate realistic photos with high fidelity from scene sketches and outperform state-of-the-art photo synthesis baselines. We also demonstrate that our framework facilitates a controllable manipulation of photo synthesis by editing strokes of corresponding sketches, delivering more fine-grained details than previous approaches that rely on region-level editing.