CityGPT: Empowering Urban Spatial Cognition of Large Language Models

TL;DR

CityGPT introduces a framework that enhances large language models' understanding of urban environments by integrating a city-scale world model, specialized instruction tuning, and a new spatial benchmark, significantly improving their geospatial reasoning in urban tasks.

Contribution

The paper presents CityGPT, a novel approach combining urban knowledge injection, a self-weighted fine-tuning method, and a comprehensive spatial benchmark to improve LLMs' urban spatial reasoning.

Findings

Smaller LLMs trained with CityGPT outperform some proprietary models on urban tasks.

The CityInstruction dataset effectively injects urban knowledge into LLMs.

The SWFT method enhances LLMs' urban spatial reasoning without sacrificing general capabilities.

Abstract

Large language models(LLMs), with their powerful language generation and reasoning capabilities, have already achieved notable success in many domains, e.g., math and code generation. However, they often fall short when tackling real-life geospatial tasks within urban environments. This limitation stems from a lack of physical world knowledge and relevant data during training. To address this gap, we propose \textit{CityGPT}, a systematic framework designed to enhance LLMs' understanding of urban space and improve their ability to solve the related urban tasks by integrating a city-scale `world model' into the model. Firstly, we construct a diverse instruction tuning dataset, \textit{CityInstruction}, for injecting urban knowledge into LLMs and effectively boosting their spatial reasoning capabilities. Using a combination of \textit{CityInstruction} and open source general instruction data, we introduce a novel and easy-to-use self-weighted fine-tuning method (\textit{SWFT}) to train various LLMs (including ChatGLM3-6B, Llama3-8B, and Qwen2.5-7B) to enhance their urban spatial capabilities without compromising, or even improving, their general abilities. Finally, to validate the effectiveness of our proposed framework, we develop a comprehensive text-based spatial benchmark \textit{CityEval} for evaluating the performance of LLMs across a wide range of urban scenarios and geospatial tasks. Extensive evaluation results demonstrate that smaller LLMs trained with \textit{CityInstruction} by \textit{SWFT} method can achieve performance that is competitive with, and in some cases superior to, proprietary LLMs when assessed using \textit{CityEval}.