Sketch2CAD: 3D CAD Model Reconstruction from 2D Sketch using Visual Transformer

TL;DR

This paper introduces a novel 3D reconstruction method from 2D sketches that produces CAD-compatible B-rep models, enabling more accurate and editable 3D models compared to traditional voxel or mesh-based methods.

Contribution

The paper proposes a visual transformer-based approach to predict scene descriptors from 2D sketches for CAD-compatible 3D reconstruction, addressing limitations of existing data formats.

Findings

Accurately reconstructs simple scenes in 3D.

Demonstrates potential for editable CAD models.

Faces challenges with complex scenes.

Abstract

Current 3D reconstruction methods typically generate outputs in the form of voxels, point clouds, or meshes. However, each of these formats has inherent limitations, such as rough surfaces and distorted structures. Additionally, these data types are not ideal for further manual editing and post-processing. In this paper, we present a novel 3D reconstruction method designed to overcome these disadvantages by reconstructing CAD-compatible models. We trained a visual transformer to predict a "scene descriptor" from a single 2D wire-frame image. This descriptor includes essential information, such as object types and parameters like position, rotation, and size. Using the predicted parameters, a 3D scene can be reconstructed with 3D modeling software that has programmable interfaces, such as Rhino Grasshopper, to build highly editable 3D models in the form of B-rep. To evaluate our proposed model, we created two datasets: one consisting of simple scenes and another with more complex scenes. The test results indicate the model's capability to accurately reconstruct simple scenes while highlighting its difficulties with more complex ones.