3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding

TL;DR

3DCity-LLM introduces a multi-modality large language model tailored for 3D city-scale perception, utilizing a new large dataset and a coarse-to-fine encoding strategy to enhance urban scene understanding.

Contribution

The paper presents a novel framework and dataset for 3D city-scale vision-language tasks, advancing large language models' capabilities in urban environment perception.

Findings

Outperforms existing state-of-the-art methods on benchmarks.

Provides a large, high-quality dataset with diverse urban scenarios.

Demonstrates improved spatial reasoning and urban understanding.

Abstract

While multi-modality large language models excel in object-centric or indoor scenarios, scaling them to 3D city-scale environments remains a formidable challenge. To bridge this gap, we propose 3DCity-LLM, a unified framework designed for 3D city-scale vision-language perception and understanding. 3DCity-LLM employs a coarse-to-fine feature encoding strategy comprising three parallel branches for target object, inter-object relationship, and global scene. To facilitate large-scale training, we introduce 3DCity-LLM-1.2M dataset that comprises approximately 1.2 million high-quality samples across seven representative task categories, ranging from fine-grained object analysis to multi-faceted scene planning. This strictly quality-controlled dataset integrates explicit 3D numerical information and diverse user-oriented simulations, enriching the question-answering diversity and realism of urban scenarios. Furthermore, we apply a multi-dimensional protocol based on text-similarity metrics and LLM-based semantic assessment to ensure faithful and comprehensive evaluations for all methods. Extensive experiments on two benchmarks demonstrate that 3DCity-LLM significantly outperforms existing state-of-the-art methods, offering a promising and meaningful direction for advancing spatial reasoning and urban intelligence. The source code and dataset are available at https://github.com/SYSU-3DSTAILab/3D-City-LLM.