CityX: Controllable Procedural Content Generation for Unbounded 3D Cities

TL;DR

CityX introduces a controllable procedural content generation framework for creating diverse, realistic, and high-fidelity 3D urban environments, aiding autonomous agent training and urban simulation.

Contribution

The paper presents a novel multi-agent, plugin-based PCG system that transforms multi-modal inputs into controllable, realistic 3D city models with visual feedback refinement.

Findings

Generates diverse, realistic 3D cities with high fidelity.

Enables real-time deployment and infinite data generation.

Outperforms existing methods in controllability and diversity.

Abstract

Urban areas, as the primary human habitat in modern civilization, accommodate a broad spectrum of social activities. With the surge of embodied intelligence, recent years have witnessed an increasing presence of physical agents in urban areas, such as autonomous vehicles and delivery robots. As a result, practitioners significantly value crafting authentic, simulation-ready 3D cities to facilitate the training and verification of such agents. However, this task is quite challenging. Current generative methods fall short in either diversity, controllability, or fidelity. In this work, we resort to the procedural content generation (PCG) technique for high-fidelity generation. It assembles superior assets according to empirical rules, ultimately leading to industrial-grade outcomes. To ensure diverse and self contained creation, we design a management protocol to accommodate extensive PCG plugins with distinct functions and interfaces. Based on this unified PCG library, we develop a multi-agent framework to transform multi-modal instructions, including OSM, semantic maps, and satellite images, into executable programs. The programs coordinate relevant plugins to construct the 3D city consistent with the control condition. A visual feedback scheme is introduced to further refine the initial outcomes. Our method, named CityX, demonstrates its superiority in creating diverse, controllable, and realistic 3D urban scenes. The synthetic scenes can be seamlessly deployed as a real-time simulator and an infinite data generator for embodied intelligence research. Our project page: https://cityx-lab.github.io.