SSEditor: Controllable Mask-to-Scene Generation with Diffusion Model

TL;DR

SSEditor is a novel 3D scene generation framework that allows controllable, target-specific scene editing without multiple resampling steps, improving flexibility and scene quality.

Contribution

It introduces a two-stage diffusion-based framework with a geometric-semantic fusion module for controllable 3D scene editing and generation.

Findings

Outperforms previous methods in controllability and quality

Capable of generating novel urban scenes on unseen datasets

Enables rapid 3D scene construction

Abstract

Recent advancements in 3D diffusion-based semantic scene generation have gained attention. However, existing methods rely on unconditional generation and require multiple resampling steps when editing scenes, which significantly limits their controllability and flexibility. To this end, we propose SSEditor, a controllable Semantic Scene Editor that can generate specified target categories without multiple-step resampling. SSEditor employs a two-stage diffusion-based framework: (1) a 3D scene autoencoder is trained to obtain latent triplane features, and (2) a mask-conditional diffusion model is trained for customizable 3D semantic scene generation. In the second stage, we introduce a geometric-semantic fusion module that enhance the model's ability to learn geometric and semantic information. This ensures that objects are generated with correct positions, sizes, and categories. Extensive experiments on SemanticKITTI and CarlaSC demonstrate that SSEditor outperforms previous approaches in terms of controllability and flexibility in target generation, as well as the quality of semantic scene generation and reconstruction. More importantly, experiments on the unseen Occ-3D Waymo dataset show that SSEditor is capable of generating novel urban scenes, enabling the rapid construction of 3D scenes.