Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Images

TL;DR

This paper introduces Neural Assembler, a model that converts multi-view images of 3D models into detailed, executable robotic assembly instructions, addressing recognition, pose estimation, and assembly sequencing.

Contribution

It presents a novel end-to-end neural model for translating multi-view images into assembly instructions, establishing benchmarks and demonstrating superior performance over alternatives.

Findings

Neural Assembler outperforms alternative methods in accuracy.

The model effectively recognizes components and estimates their poses.

Benchmarks validate the approach's robustness and applicability.

Abstract

Image-guided object assembly represents a burgeoning research topic in computer vision. This paper introduces a novel task: translating multi-view images of a structural 3D model (for example, one constructed with building blocks drawn from a 3D-object library) into a detailed sequence of assembly instructions executable by a robotic arm. Fed with multi-view images of the target 3D model for replication, the model designed for this task must address several sub-tasks, including recognizing individual components used in constructing the 3D model, estimating the geometric pose of each component, and deducing a feasible assembly order adhering to physical rules. Establishing accurate 2D-3D correspondence between multi-view images and 3D objects is technically challenging. To tackle this, we propose an end-to-end model known as the Neural Assembler. This model learns an object graph where each vertex represents recognized components from the images, and the edges specify the topology of the 3D model, enabling the derivation of an assembly plan. We establish benchmarks for this task and conduct comprehensive empirical evaluations of Neural Assembler and alternative solutions. Our experiments clearly demonstrate the superiority of Neural Assembler.