CityCraft: A Real Crafter for 3D City Generation

TL;DR

CityCraft introduces a novel framework for generating diverse, realistic 3D city scenes by combining diffusion transformers, language-guided planning, and asset retrieval, advancing urban scene synthesis for applications like autonomous driving and city planning.

Contribution

The paper presents a new multi-stage approach integrating diffusion transformers, language models, and asset placement, along with two datasets, to improve diversity and realism in 3D city generation.

Findings

Achieves state-of-the-art realism in 3D city generation

Provides controllable and diverse urban scene layouts

Introduces two new datasets for urban scene synthesis

Abstract

City scene generation has gained significant attention in autonomous driving, smart city development, and traffic simulation. It helps enhance infrastructure planning and monitoring solutions. Existing methods have employed a two-stage process involving city layout generation, typically using Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), or Transformers, followed by neural rendering. These techniques often exhibit limited diversity and noticeable artifacts in the rendered city scenes. The rendered scenes lack variety, resembling the training images, resulting in monotonous styles. Additionally, these methods lack planning capabilities, leading to less realistic generated scenes. In this paper, we introduce CityCraft, an innovative framework designed to enhance both the diversity and quality of urban scene generation. Our approach integrates three key stages: initially, a diffusion transformer (DiT) model is deployed to generate diverse and controllable 2D city layouts. Subsequently, a Large Language Model(LLM) is utilized to strategically make land-use plans within these layouts based on user prompts and language guidelines. Based on the generated layout and city plan, we utilize the asset retrieval module and Blender for precise asset placement and scene construction. Furthermore, we contribute two new datasets to the field: 1)CityCraft-OSM dataset including 2D semantic layouts of urban areas, corresponding satellite images, and detailed annotations. 2) CityCraft-Buildings dataset, featuring thousands of diverse, high-quality 3D building assets. CityCraft achieves state-of-the-art performance in generating realistic 3D cities.