Co-generation of Layout and Shape from Text via Autoregressive 3D Diffusion

TL;DR

This paper introduces a novel autoregressive diffusion model for sequential text-to-3D scene generation, enabling detailed and semantically consistent indoor scene creation from textual descriptions.

Contribution

It proposes 3D-ARD+, a unified generative model that combines autoregressive token sequence modeling with diffusion-based 3D object generation, advancing the realism and accuracy of scene synthesis.

Findings

The model can generate complex indoor scenes with non-trivial spatial arrangements.

It outperforms existing methods in generating semantically consistent 3D scenes.

A large dataset of 230K indoor scenes was curated for training.

Abstract

Recent text-to-scene generation approaches largely reduced the manual efforts required to create 3D scenes. However, their focus is either to generate a scene layout or to generate objects, and few generate both. The generated scene layout is often simple even with LLM's help. Moreover, the generated scene is often inconsistent with the text input that contains non-trivial descriptions of the shape, appearance, and spatial arrangement of the objects. We present a new paradigm of sequential text-to-scene generation and propose a novel generative model for interactive scene creation. At the core is a 3D Autoregressive Diffusion model 3D-ARD+, which unifies the autoregressive generation over a multimodal token sequence and diffusion generation of next-object 3D latents. To generate the next object, the model uses one autoregressive step to generate the coarse-grained 3D latents in the scene space, conditioned on both the current seen text instructions and already synthesized 3D scene. It then uses a second step to generate the 3D latents in the smaller object space, which can be decoded into fine-grained object geometry and appearance. We curate a large dataset of 230K indoor scenes with paired text instructions for training. We evaluate 7B 3D-ARD+, on challenging scenes, and showcase the model can generate and place objects following non-trivial spatial layout and semantics prescribed by the text instructions.