BrickCraft: Visuomotor Skill Composition with Situated Manual Guidance for Long-Horizon Interlocking Brick Assembly

TL;DR

BrickCraft is a framework enabling robots to perform complex, long-horizon interlocking brick assembly by decomposing tasks into reusable skills guided by situated manuals, demonstrating strong generalization from limited demonstrations.

Contribution

The paper introduces a compositional framework that integrates spatially grounded visuomotor skills with situated manuals for generalizable long-horizon brick assembly.

Findings

Proficient assembly skills learned from limited demonstrations.

Strong generalization to unseen structures demonstrated.

Effective bridging of high-level plans and physical execution.

Abstract

Autonomous robotic assembly of interlocking bricks demands seamless integration of long-horizon task reasoning, spatial grounding, and fine-grained manipulation. This paper presents BrickCraft, a compositional framework designed for long-horizon and generalizable interlocking brick assembly. BrickCraft models the assembly process using a relative formulation, where each step is anchored to a reference brick within the partial structure, thereby decomposing complex tasks into a finite set of reusable primitive skills. BrickCraft bridges the gap between high-level assembly plans and physical execution through situated manuals, which provide explicit spatial guidance for learned visuomotor skills by projecting the assembly intent onto real-time robot observations. Finally, BrickCraft employs a compositional execution pipeline that chains these spatially grounded skills to accomplish long-horizon assembly tasks. Extensive experimental validations demonstrate that BrickCraft acquires proficient assembly skills from a limited set of demonstrations and exhibits strong compositional generalization to unseen structures. The project website is available at https://intelligent-control-lab.github.io/BrickCraft.