Translating a Visual LEGO Manual to a Machine-Executable Plan

TL;DR

This paper introduces MEPNet, a learning-based framework that translates visual LEGO manuals into machine-executable assembly plans by predicting component placements and poses, handling unseen parts and establishing 2D-3D correspondence.

Contribution

The paper presents a novel neural network architecture, MEPNet, that effectively converts image-based assembly instructions into 3D plans, outperforming existing methods on multiple datasets.

Findings

MEPNet achieves higher accuracy than previous methods.

The framework generalizes well to unseen components.

It effectively establishes 2D-3D correspondence for assembly tasks.

Abstract

We study the problem of translating an image-based, step-by-step assembly manual created by human designers into machine-interpretable instructions. We formulate this problem as a sequential prediction task: at each step, our model reads the manual, locates the components to be added to the current shape, and infers their 3D poses. This task poses the challenge of establishing a 2D-3D correspondence between the manual image and the real 3D object, and 3D pose estimation for unseen 3D objects, since a new component to be added in a step can be an object built from previous steps. To address these two challenges, we present a novel learning-based framework, the Manual-to-Executable-Plan Network (MEPNet), which reconstructs the assembly steps from a sequence of manual images. The key idea is to integrate neural 2D keypoint detection modules and 2D-3D projection algorithms for high-precision prediction and strong generalization to unseen components. The MEPNet outperforms existing methods on three newly collected LEGO manual datasets and a Minecraft house dataset.