ScenDi: 3D-to-2D Scene Diffusion Cascades for Urban Generation

TL;DR

ScenDi combines 3D and 2D diffusion models to generate realistic, controllable urban scenes with detailed appearance, addressing limitations of previous methods in detail preservation and camera control.

Contribution

We introduce a novel framework that integrates 3D latent diffusion and 2D video diffusion models for urban scene generation with controllability and enhanced detail.

Findings

Effective generation of urban scenes on Waymo and KITTI-360 datasets.

Ability to control scenes via 3D bounding boxes, road maps, or text prompts.

Scenes adhere accurately to specified camera trajectories.

Abstract

Recent advancements in 3D object generation using diffusion models have achieved remarkable success, but generating realistic 3D urban scenes remains challenging. Existing methods relying solely on 3D diffusion models tend to suffer a degradation in appearance details, while those utilizing only 2D diffusion models typically compromise camera controllability. To overcome this limitation, we propose ScenDi, a method for urban scene generation that integrates both 3D and 2D diffusion models. We first train a 3D latent diffusion model to generate 3D Gaussians, enabling the rendering of images at a relatively low resolution. To enable controllable synthesis, this 3DGS generation process can be optionally conditioned by specifying inputs such as 3d bounding boxes, road maps, or text prompts. Then, we train a 2D video diffusion model to enhance appearance details conditioned on rendered images from the 3D Gaussians. By leveraging the coarse 3D scene as guidance for 2D video diffusion, ScenDi generates desired scenes based on input conditions and successfully adheres to accurate camera trajectories. Experiments on two challenging real-world datasets, Waymo and KITTI-360, demonstrate the effectiveness of our approach.