PASTA: Controllable Part-Aware Shape Generation with Autoregressive Transformers

TL;DR

PASTA is a novel autoregressive transformer framework that generates high-quality 3D shapes by modeling part-based structures, enabling diverse, realistic, and controllable shape synthesis from various inputs.

Contribution

The paper introduces PASTA, a new part-aware autoregressive transformer architecture for 3D shape generation, combining primitive-based modeling with a blending network for high-quality mesh synthesis.

Findings

Outperforms existing methods in realism and diversity of generated shapes.

Capable of generating shapes from scratch, partial inputs, text, and images.

Enables part-specific shape variation and size-guided generation.

Abstract

The increased demand for tools that automate the 3D content creation process led to tremendous progress in deep generative models that can generate diverse 3D objects of high fidelity. In this paper, we present PASTA, an autoregressive transformer architecture for generating high quality 3D shapes. PASTA comprises two main components: An autoregressive transformer that generates objects as a sequence of cuboidal primitives and a blending network, implemented with a transformer decoder that composes the sequences of cuboids and synthesizes high quality meshes for each object. Our model is trained in two stages: First we train our autoregressive generative model using only annotated cuboidal parts as supervision and next, we train our blending network using explicit 3D supervision, in the form of watertight meshes. Evaluations on various ShapeNet objects showcase the ability of our model to perform shape generation from diverse inputs \eg from scratch, from a partial object, from text and images, as well size-guided generation, by explicitly conditioning on a bounding box that defines the object's boundaries. Moreover, as our model considers the underlying part-based structure of a 3D object, we are able to select a specific part and produce shapes with meaningful variations of this part. As evidenced by our experiments, our model generates 3D shapes that are both more realistic and diverse than existing part-based and non part-based methods, while at the same time is simpler to implement and train.