Learning to Build by Building Your Own Instructions
Aaron Walsman, Muru Zhang, Adam Fishman, Ali Farhadi, Dieter Fox

TL;DR
This paper introduces \\ours, a novel agent that learns to build complex LEGO structures by creating visual instructions through disassembly, enabling reasoning and training on larger assemblies than before.
Contribution
The paper presents a new method for learning to build complex objects by disassembling and creating explicit visual instructions, advancing the capabilities in the Break-and-Make problem.
Findings
The model can successfully disassemble and reassemble LEGO structures with over 100 steps.
It enables training on larger assemblies than previous methods.
The approach improves learning efficiency and usability in the Break-and-Make setting.
Abstract
Structural understanding of complex visual objects is an important unsolved component of artificial intelligence. To study this, we develop a new technique for the recently proposed Break-and-Make problem in LTRON where an agent must learn to build a previously unseen LEGO assembly using a single interactive session to gather information about its components and their structure. We attack this problem by building an agent that we call \textbf{\ours} that is able to make its own visual instruction book. By disassembling an unseen assembly and periodically saving images of it, the agent is able to create a set of instructions so that it has the information necessary to rebuild it. These instructions form an explicit memory that allows the model to reason about the assembly process one step at a time, avoiding the need for long-term implicit memory. This in turn allows us to train on much…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsEducation and Technology Integration
MethodsSparse Evolutionary Training
