3D VR Sketch Guided 3D Shape Prototyping and Exploration

TL;DR

This paper introduces a novel 3D shape generation network conditioned on VR sketches, enabling novices to create realistic 3D models efficiently, with multiple shape outputs to handle sketch ambiguity.

Contribution

It presents a new 3D shape generation method that uses VR sketches as input, supports limited data training, and ensures realistic and faithful shape reconstruction.

Findings

Supports multiple shape outputs for ambiguous sketches

Leverages normalizing flow for realistic shape generation

Effective training with limited data

Abstract

3D shape modeling is labor-intensive, time-consuming, and requires years of expertise. To facilitate 3D shape modeling, we propose a 3D shape generation network that takes a 3D VR sketch as a condition. We assume that sketches are created by novices without art training and aim to reconstruct geometrically realistic 3D shapes of a given category. To handle potential sketch ambiguity, our method creates multiple 3D shapes that align with the original sketch's structure. We carefully design our method, training the model step-by-step and leveraging multi-modal 3D shape representation to support training with limited training data. To guarantee the realism of generated 3D shapes we leverage the normalizing flow that models the distribution of the latent space of 3D shapes. To encourage the fidelity of the generated 3D shapes to an input sketch, we propose a dedicated loss that we deploy at different stages of the training process. The code is available at https://github.com/Rowl1ng/3Dsketch2shape.