What can LLM tell us about cities?

TL;DR

This paper investigates how large language models can provide knowledge about global cities, revealing their strengths and limitations in city-related information and their potential for data-driven urban decision-making.

Contribution

It introduces methods to extract city knowledge from LLMs and demonstrates their effectiveness in improving predictive models for urban data analysis.

Findings

LLMs contain broad city-related knowledge with varying accuracy.

ML models using LLM-derived features outperform traditional models.

LLMs generate generic outputs when lacking specific city knowledge.

Abstract

This study explores the capabilities of large language models (LLMs) in providing knowledge about cities and regions on a global scale. We employ two methods: directly querying the LLM for target variable values and extracting explicit and implicit features from the LLM correlated with the target variable. Our experiments reveal that LLMs embed a broad but varying degree of knowledge across global cities, with ML models trained on LLM-derived features consistently leading to improved predictive accuracy. Additionally, we observe that LLMs demonstrate a certain level of knowledge across global cities on all continents, but it is evident when they lack knowledge, as they tend to generate generic or random outputs for unfamiliar tasks. These findings suggest that LLMs can offer new opportunities for data-driven decision-making in the study of cities.