GenAI Models Capture Urban Science but Oversimplify Complexity

TL;DR

This paper evaluates how well GenAI models generate urban science data, revealing they capture core theories but oversimplify complexity, with proposed calibration improving data fidelity.

Contribution

Introduces AI4US framework for systematic evaluation of GenAI in urban science, highlighting limitations and proposing a calibration method to enhance data quality.

Findings

GenAI models reproduce core urban theories but lack diversity.

Models show systematic deviations in generated data.

Calibration with optimal transport improves data fidelity.

Abstract

Generative artificial intelligence (GenAI) models are increasingly used for scientific data generation, yet their alignment with empirical knowledge in urban science remains unclear. Here, we introduce AI4US (Artificial Intelligence for Urban Science), a framework that systematically evaluates leading GenAI models by testing their fidelity in generating both symbolic and perceptual urban data. For the symbolic domain, we benchmark generated data against foundational urban theories concerning scale, space, and morphology. For the perceptual domain, we validate the models' visual judgments against human benchmarks and, critically, leverage their generative control to conduct in causal experiments on urban perception. Our findings show that while GenAI models reproduce core theoretical patterns, the generated data exhibit crucial limitations: poor diversity, systematic parametric deviations, and improvement from prompt engineering. To address this, we introduce a post-hoc calibration procedure using optimal transport, which produces synthetic symbolic datasets with demonstrably higher fidelity.