Unsupervised Scene Sketch to Photo Synthesis
Jiayun Wang, Sangryul Jeon, Stella X. Yu, Xi Zhang, Himanshu Arora, Yu, Lou

TL;DR
This paper introduces an unsupervised method to generate realistic photos from scene sketches by standardizing data and disentangling structure and style, enabling high-fidelity photo synthesis and controllable editing.
Contribution
The method uniquely leverages a standardization module to create pseudo pairs and disentangles structure and style for flexible, high-quality photo synthesis from sketches without paired data.
Findings
Outperforms state-of-the-art photo synthesis methods in perceptual metrics.
Enables fine-grained, stroke-level editing of synthesized photos.
Produces realistic images with high fidelity from scene sketches.
Abstract
Sketches make an intuitive and powerful visual expression as they are fast executed freehand drawings. We present a method for synthesizing realistic photos from scene sketches. Without the need for sketch and photo pairs, our framework directly learns from readily available large-scale photo datasets in an unsupervised manner. To this end, we introduce a standardization module that provides pseudo sketch-photo pairs during training by converting photos and sketches to a standardized domain, i.e. the edge map. The reduced domain gap between sketch and photo also allows us to disentangle them into two components: holistic scene structures and low-level visual styles such as color and texture. Taking this advantage, we synthesize a photo-realistic image by combining the structure of a sketch and the visual style of a reference photo. Extensive experimental results on perceptual similarity…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Vision and Imaging · Advanced Image and Video Retrieval Techniques · Generative Adversarial Networks and Image Synthesis
