BrickNet: Graph-Backed Generative Brick Assembly

TL;DR

BrickNet introduces a graph-based generative model for LEGO brick assembly, handling diverse parts and complex structures, supported by a large dataset and accessible code for research.

Contribution

The paper presents a novel graph-backed language model for LEGO assembly that manages diverse pieces and complex constraints, expanding beyond prior voxel-based methods.

Findings

Successfully generated valid LEGO structures with diverse parts.

Large-scale dataset of 100,000+ human-designed LEGO objects created.

Model and dataset made publicly available for research.

Abstract

We train a language model to generate LEGO-brick build sequences. While prior work has been restricted to discrete, voxel-like towers, we consider a much broader set of pieces, encompassing thousands of part types with diverse connection semantics. To enable this, we first collect a large-scale dataset of over 100,000 human-designed LDraw brick objects and scenes. The complexity of our setting makes it challenging to autoregressively assemble structures that satisfy physical constraints. When predicting block pose directly, build sequences quickly become invalid after a small number of steps. Although pieces are placed in 3D space, it is the spatial relationships of the parts which define the whole. With this in mind, we design a graph-based program representation that parametrizes structure through connectivity, improving the physical grounding of generated sequences. To enable future applications, we make our dataset and models available for research purposes. https://kulits.github.io/BrickNet