LT3SD: Latent Trees for 3D Scene Diffusion

TL;DR

LT3SD introduces a latent tree-based diffusion model that effectively generates large-scale, detailed 3D scenes by hierarchical encoding and patch-based synthesis, advancing 3D scene generation quality and scalability.

Contribution

The paper proposes a novel latent tree representation and diffusion process for large-scale 3D scene generation, enabling high-quality, diverse, and scalable scene synthesis.

Findings

Effective large-scale scene generation demonstrated

High-quality scene completion achieved

Scalable patch-based synthesis method validated

Abstract

We present LT3SD, a novel latent diffusion model for large-scale 3D scene generation. Recent advances in diffusion models have shown impressive results in 3D object generation, but are limited in spatial extent and quality when extended to 3D scenes. To generate complex and diverse 3D scene structures, we introduce a latent tree representation to effectively encode both lower-frequency geometry and higher-frequency detail in a coarse-to-fine hierarchy. We can then learn a generative diffusion process in this latent 3D scene space, modeling the latent components of a scene at each resolution level. To synthesize large-scale scenes with varying sizes, we train our diffusion model on scene patches and synthesize arbitrary-sized output 3D scenes through shared diffusion generation across multiple scene patches. Through extensive experiments, we demonstrate the efficacy and benefits of LT3SD for large-scale, high-quality unconditional 3D scene generation and for probabilistic completion for partial scene observations.