SceneCraft: Layout-Guided 3D Scene Generation

TL;DR

SceneCraft is a novel approach that generates detailed, multi-room indoor 3D scenes from textual descriptions and layout preferences, using a rendering-based diffusion model and neural radiance fields to produce realistic, complex environments.

Contribution

It introduces a new method combining semantic layout conversion, diffusion models, and NeRFs for large-scale indoor scene generation beyond previous small-scale limitations.

Findings

Outperforms existing methods in complex indoor scene generation

Supports multi-room environments with irregular shapes and layouts

Produces high-quality, realistic visual results

Abstract

The creation of complex 3D scenes tailored to user specifications has been a tedious and challenging task with traditional 3D modeling tools. Although some pioneering methods have achieved automatic text-to-3D generation, they are generally limited to small-scale scenes with restricted control over the shape and texture. We introduce SceneCraft, a novel method for generating detailed indoor scenes that adhere to textual descriptions and spatial layout preferences provided by users. Central to our method is a rendering-based technique, which converts 3D semantic layouts into multi-view 2D proxy maps. Furthermore, we design a semantic and depth conditioned diffusion model to generate multi-view images, which are used to learn a neural radiance field (NeRF) as the final scene representation. Without the constraints of panorama image generation, we surpass previous methods in supporting complicated indoor space generation beyond a single room, even as complicated as a whole multi-bedroom apartment with irregular shapes and layouts. Through experimental analysis, we demonstrate that our method significantly outperforms existing approaches in complex indoor scene generation with diverse textures, consistent geometry, and realistic visual quality. Code and more results are available at: https://orangesodahub.github.io/SceneCraft