SPAFormer: Sequential 3D Part Assembly with Transformers

TL;DR

SPAFormer is a novel transformer-based model that effectively addresses the combinatorial complexity in 3D part assembly tasks by leveraging sequence constraints and knowledge enhancement, demonstrating superior generalization on a new challenging benchmark.

Contribution

The paper introduces SPAFormer, a transformer model that reduces solution space complexity in 3D-PA by using sequence constraints and knowledge strategies, with new benchmark PartNet-Assembly.

Findings

Outperforms existing methods in generalization and long-horizon assembly tasks.

Effectively reduces combinatorial explosion in 3D part assembly.

Demonstrates strong results on the new PartNet-Assembly benchmark.

Abstract

We introduce SPAFormer, an innovative model designed to overcome the combinatorial explosion challenge in the 3D Part Assembly (3D-PA) task. This task requires accurate prediction of each part's poses in sequential steps. As the number of parts increases, the possible assembly combinations increase exponentially, leading to a combinatorial explosion that severely hinders the efficacy of 3D-PA. SPAFormer addresses this problem by leveraging weak constraints from assembly sequences, effectively reducing the solution space's complexity. Since the sequence of parts conveys construction rules similar to sentences structured through words, our model explores both parallel and autoregressive generation. We further strengthen SPAFormer through knowledge enhancement strategies that utilize the attributes of parts and their sequence information, enabling it to capture the inherent assembly pattern and relationships among sequentially ordered parts. We also construct a more challenging benchmark named PartNet-Assembly covering 21 varied categories to more comprehensively validate the effectiveness of SPAFormer. Extensive experiments demonstrate the superior generalization capabilities of SPAFormer, particularly with multi-tasking and in scenarios requiring long-horizon assembly. Code is available at https://github.com/xuboshen/SPAFormer.