Deep3DSketch+: Rapid 3D Modeling from Single Free-hand Sketches

TL;DR

Deep3DSketch+ is an end-to-end method that enables rapid, high-fidelity 3D modeling from a single free-hand sketch, making 3D content creation more accessible and efficient for AR/VR applications.

Contribution

It introduces a lightweight network and a structural-aware adversarial training approach with SEM for single-sketch 3D modeling, achieving state-of-the-art results.

Findings

Effective real-time inference demonstrated.

High-fidelity 3D shapes from single sketches.

Outperforms existing methods on synthetic and real datasets.

Abstract

The rapid development of AR/VR brings tremendous demands for 3D content. While the widely-used Computer-Aided Design (CAD) method requires a time-consuming and labor-intensive modeling process, sketch-based 3D modeling offers a potential solution as a natural form of computer-human interaction. However, the sparsity and ambiguity of sketches make it challenging to generate high-fidelity content reflecting creators' ideas. Precise drawing from multiple views or strategic step-by-step drawings is often required to tackle the challenge but is not friendly to novice users. In this work, we introduce a novel end-to-end approach, Deep3DSketch+, which performs 3D modeling using only a single free-hand sketch without inputting multiple sketches or view information. Specifically, we introduce a lightweight generation network for efficient inference in real-time and a structural-aware adversarial training approach with a Stroke Enhancement Module (SEM) to capture the structural information to facilitate learning of the realistic and fine-detailed shape structures for high-fidelity performance. Extensive experiments demonstrated the effectiveness of our approach with the state-of-the-art (SOTA) performance on both synthetic and real datasets.