Break and Make: Interactive Structural Understanding Using LEGO Bricks

TL;DR

This paper introduces the 'Break and Make' problem, a new interactive task where agents understand and reconstruct complex LEGO models through disassembly and assembly, supported by a new simulator and dataset.

Contribution

It proposes a novel interactive reasoning problem for structural understanding, along with a 3D simulator and a large LEGO dataset for research and benchmarking.

Findings

Initial sequence-to-sequence models provide guidance for the task.

The LTRON simulator enables complex LEGO manipulation tasks.

A new dataset of over a thousand LEGO scenes is introduced.

Abstract

Visual understanding of geometric structures with complex spatial relationships is a fundamental component of human intelligence. As children, we learn how to reason about structure not only from observation, but also by interacting with the world around us -- by taking things apart and putting them back together again. The ability to reason about structure and compositionality allows us to not only build things, but also understand and reverse-engineer complex systems. In order to advance research in interactive reasoning for part-based geometric understanding, we propose a challenging new assembly problem using LEGO bricks that we call Break and Make. In this problem an agent is given a LEGO model and attempts to understand its structure by interactively inspecting and disassembling it. After this inspection period, the agent must then prove its understanding by rebuilding the model from scratch using low-level action primitives. In order to facilitate research on this problem we have built LTRON, a fully interactive 3D simulator that allows learning agents to assemble, disassemble and manipulate LEGO models. We pair this simulator with a new dataset of fan-made LEGO creations that have been uploaded to the internet in order to provide complex scenes containing over a thousand unique brick shapes. We take a first step towards solving this problem using sequence-to-sequence models that provide guidance for how to make progress on this challenging problem. Our simulator and data are available at github.com/aaronwalsman/ltron. Additional training code and PyTorch examples are available at github.com/aaronwalsman/ltron-torch-eccv22.