AutoPartGen: Autogressive 3D Part Generation and Discovery

TL;DR

AutoPartGen is a novel autoregressive model that generates 3D objects by sequentially predicting parts conditioned on various inputs, enabling high-quality, compositional 3D reconstructions with automatic part determination.

Contribution

It introduces AutoPartGen, the first autoregressive 3D part generation model that leverages a powerful latent space for compositional object synthesis from diverse inputs.

Findings

Achieves state-of-the-art results in 3D part generation

Effectively generates coherent 3D objects from images, masks, or existing 3D models

Automatically determines the number and type of parts during generation

Abstract

We introduce AutoPartGen, a model that generates objects composed of 3D parts in an autoregressive manner. This model can take as input an image of an object, 2D masks of the object's parts, or an existing 3D object, and generate a corresponding compositional 3D reconstruction. Our approach builds upon 3DShape2VecSet, a recent latent 3D representation with powerful geometric expressiveness. We observe that this latent space exhibits strong compositional properties, making it particularly well-suited for part-based generation tasks. Specifically, AutoPartGen generates object parts autoregressively, predicting one part at a time while conditioning on previously generated parts and additional inputs, such as 2D images, masks, or 3D objects. This process continues until the model decides that all parts have been generated, thus determining automatically the type and number of parts. The resulting parts can be seamlessly assembled into coherent objects or scenes without requiring additional optimization. We evaluate both the overall 3D generation capabilities and the part-level generation quality of AutoPartGen, demonstrating that it achieves state-of-the-art performance in 3D part generation.