CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks

TL;DR

CityBench is a comprehensive evaluation platform for assessing large language and vision-language models on diverse urban tasks, highlighting their strengths in understanding urban semantics and their limitations in professional and numerical tasks.

Contribution

This paper introduces CityBench, the first systematic benchmark for evaluating LLMs and VLMs in urban research, integrating urban data and simulation for diverse task assessment.

Findings

LLMs perform well on semantic and commonsense urban tasks.

LLMs struggle with geospatial prediction and traffic control.

Advanced models show competitive performance in urban understanding.

Abstract

As large language models (LLMs) continue to advance and gain widespread use, establishing systematic and reliable evaluation methodologies for LLMs and vision-language models (VLMs) has become essential to ensure their real-world effectiveness and reliability. There have been some early explorations about the usability of LLMs for limited urban tasks, but a systematic and scalable evaluation benchmark is still lacking. The challenge in constructing a systematic evaluation benchmark for urban research lies in the diversity of urban data, the complexity of application scenarios and the highly dynamic nature of the urban environment. In this paper, we design \textit{CityBench}, an interactive simulator based evaluation platform, as the first systematic benchmark for evaluating the capabilities of LLMs for diverse tasks in urban research. First, we build \textit{CityData} to integrate the diverse urban data and \textit{CitySimu} to simulate fine-grained urban dynamics. Based on \textit{CityData} and \textit{CitySimu}, we design 8 representative urban tasks in 2 categories of perception-understanding and decision-making as the \textit{CityBench}. With extensive results from 30 well-known LLMs and VLMs in 13 cities around the world, we find that advanced LLMs and VLMs can achieve competitive performance in diverse urban tasks requiring commonsense and semantic understanding abilities, e.g., understanding the human dynamics and semantic inference of urban images. Meanwhile, they fail to solve the challenging urban tasks requiring professional knowledge and high-level numerical abilities, e.g., geospatial prediction and traffic control task.