SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation

TL;DR

SceneFactor introduces a diffusion-based method for large-scale, controllable 3D scene generation and editing by leveraging factored semantic and geometric manifolds, enabling intuitive manipulation of generated scenes.

Contribution

The paper proposes a novel factored diffusion formulation that allows for controllable and editable 3D scene synthesis using semantic 3D boxes as proxies.

Findings

Enables text-guided 3D scene synthesis with controllability.

Allows intuitive, localized editing of 3D scenes via semantic proxies.

Demonstrates high-fidelity 3D scene generation with effective editing capabilities.

Abstract

We present SceneFactor, a diffusion-based approach for large-scale 3D scene generation that enables controllable generation and effortless editing. SceneFactor enables text-guided 3D scene synthesis through our factored diffusion formulation, leveraging latent semantic and geometric manifolds for generation of arbitrary-sized 3D scenes. While text input enables easy, controllable generation, text guidance remains imprecise for intuitive, localized editing and manipulation of the generated 3D scenes. Our factored semantic diffusion generates a proxy semantic space composed of semantic 3D boxes that enables controllable editing of generated scenes by adding, removing, changing the size of the semantic 3D proxy boxes that guides high-fidelity, consistent 3D geometric editing. Extensive experiments demonstrate that our approach enables high-fidelity 3D scene synthesis with effective controllable editing through our factored diffusion approach.