PlankAssembly: Robust 3D Reconstruction from Three Orthographic Views with Learnt Shape Programs

TL;DR

This paper introduces PlankAssembly, a Transformer-based method that converts three orthographic 2D views into accurate 3D CAD models by learning flexible mappings and using shape programs, especially effective with noisy or incomplete inputs.

Contribution

It presents a novel Transformer-based approach with shape programs for robust 3D reconstruction from orthographic views, improving accuracy under imperfect input conditions.

Findings

Outperforms existing methods on a new benchmark dataset

Handles noisy and incomplete input drawings effectively

Leverages attention mechanisms for flexible input-output mapping

Abstract

In this paper, we develop a new method to automatically convert 2D line drawings from three orthographic views into 3D CAD models. Existing methods for this problem reconstruct 3D models by back-projecting the 2D observations into 3D space while maintaining explicit correspondence between the input and output. Such methods are sensitive to errors and noises in the input, thus often fail in practice where the input drawings created by human designers are imperfect. To overcome this difficulty, we leverage the attention mechanism in a Transformer-based sequence generation model to learn flexible mappings between the input and output. Further, we design shape programs which are suitable for generating the objects of interest to boost the reconstruction accuracy and facilitate CAD modeling applications. Experiments on a new benchmark dataset show that our method significantly outperforms existing ones when the inputs are noisy or incomplete.