S3D: Sketch-Driven 3D Model Generation

TL;DR

S3D is a new framework that converts simple sketches into detailed 3D models using a U-Net architecture and style-alignment loss, improving reconstruction quality and robustness.

Contribution

The paper introduces S3D, a novel sketch-to-3D generation method with a style-alignment loss and data augmentation for better fidelity and robustness.

Findings

Effective 3D model generation from sketches demonstrated

Style-alignment loss improves reconstruction fidelity

Data augmentation enhances robustness

Abstract

Generating high-quality 3D models from 2D sketches is a challenging task due to the inherent ambiguity and sparsity of sketch data. In this paper, we present S3D, a novel framework that converts simple hand-drawn sketches into detailed 3D models. Our method utilizes a U-Net-based encoder-decoder architecture to convert sketches into face segmentation masks, which are then used to generate a 3D representation that can be rendered from novel views. To ensure robust consistency between the sketch domain and the 3D output, we introduce a novel style-alignment loss that aligns the U-Net bottleneck features with the initial encoder outputs of the 3D generation module, significantly enhancing reconstruction fidelity. To further enhance the network's robustness, we apply augmentation techniques to the sketch dataset. This streamlined framework demonstrates the effectiveness of S3D in generating high-quality 3D models from sketch inputs. The source code for this project is publicly available at https://github.com/hailsong/S3D.