TreeSBA: Tree-Transformer for Self-Supervised Sequential Brick Assembly

TL;DR

TreeSBA introduces a class-agnostic, transformer-based framework for sequential 3D brick assembly from images, leveraging synthetic data and self-supervision to outperform existing methods without requiring real-world annotations.

Contribution

The paper proposes a novel BFS LEGO-Tree structure and a class-agnostic tree-transformer model for efficient, annotation-free sequential brick assembly from multi-view images.

Findings

Outperforms existing 3D supervised methods by 7.8% and 11.3% in mIoU.

Uses synthetic-to-real transfer learning with self-supervision.

Achieves accurate sequential assembly without real data annotations.

Abstract

Inferring step-wise actions to assemble 3D objects with primitive bricks from images is a challenging task due to complex constraints and the vast number of possible combinations. Recent studies have demonstrated promising results on sequential LEGO brick assembly through the utilization of LEGO-Graph modeling to predict sequential actions. However, existing approaches are class-specific and require significant computational and 3D annotation resources. In this work, we first propose a computationally efficient breadth-first search (BFS) LEGO-Tree structure to model the sequential assembly actions by considering connections between consecutive layers. Based on the LEGO-Tree structure, we then design a class-agnostic tree-transformer framework to predict the sequential assembly actions from the input multi-view images. A major challenge of the sequential brick assembly task is that the step-wise action labels are costly and tedious to obtain in practice. We mitigate this problem by leveraging synthetic-to-real transfer learning. Specifically, our model is first pre-trained on synthetic data with full supervision from the available action labels. We then circumvent the requirement for action labels in the real data by proposing an action-to-silhouette projection that replaces action labels with input image silhouettes for self-supervision. Without any annotation on the real data, our model outperforms existing methods with 3D supervision by 7.8% and 11.3% in mIoU on the MNIST and ModelNet Construction datasets, respectively.