3D Reconstruction from Sketches

TL;DR

This paper presents a pipeline for 3D scene reconstruction from sketches, combining sketch stitching, image translation via CycleGAN, and depth estimation with MegaDepth, supported by a new dataset of sketch-image pairs.

Contribution

It introduces a novel dataset of sketch-image pairs and a pipeline that effectively reconstructs 3D scenes from sketches using deep learning techniques.

Findings

Pipeline performs well on diverse sketches

CycleGAN-based image conversion is effective

Stitching does not generalize well to real sketches

Abstract

We consider the problem of reconstructing a 3D scene from multiple sketches. We propose a pipeline which involves (1) stitching together multiple sketches through use of correspondence points, (2) converting the stitched sketch into a realistic image using a CycleGAN, and (3) estimating that image's depth-map using a pre-trained convolutional neural network based architecture called MegaDepth. Our contribution includes constructing a dataset of image-sketch pairs, the images for which are from the Zurich Building Database, and sketches have been generated by us. We use this dataset to train a CycleGAN for our pipeline's second step. We end up with a stitching process that does not generalize well to real drawings, but the rest of the pipeline that creates a 3D reconstruction from a single sketch performs quite well on a wide variety of drawings.