GeoProg3D: Compositional Visual Reasoning for City-Scale 3D Language Fields

TL;DR

GeoProg3D introduces a scalable, city-scale 3D language reasoning framework that combines hierarchical 3D models, geographic information, and large language models to enable natural language interactions and reasoning in complex urban environments.

Contribution

It presents the first framework for compositional geographic reasoning in high-fidelity city-scale 3D scenes using natural language and large language models.

Findings

Outperforms existing 3D language and vision-language models on GeoEval3D benchmark.

Effectively handles large-scale urban data with hierarchical 3D models and geographic filtering.

Supports diverse geographic reasoning tasks like grounding, spatial reasoning, and counting.

Abstract

The advancement of 3D language fields has enabled intuitive interactions with 3D scenes via natural language. However, existing approaches are typically limited to small-scale environments, lacking the scalability and compositional reasoning capabilities necessary for large, complex urban settings. To overcome these limitations, we propose GeoProg3D, a visual programming framework that enables natural language-driven interactions with city-scale high-fidelity 3D scenes. GeoProg3D consists of two key components: (i) a Geography-aware City-scale 3D Language Field (GCLF) that leverages a memory-efficient hierarchical 3D model to handle large-scale data, integrated with geographic information for efficiently filtering vast urban spaces using directional cues, distance measurements, elevation data, and landmark references; and (ii) Geographical Vision APIs (GV-APIs), specialized geographic vision tools such as area segmentation and object detection. Our framework employs large language models (LLMs) as reasoning engines to dynamically combine GV-APIs and operate GCLF, effectively supporting diverse geographic vision tasks. To assess performance in city-scale reasoning, we introduce GeoEval3D, a comprehensive benchmark dataset containing 952 query-answer pairs across five challenging tasks: grounding, spatial reasoning, comparison, counting, and measurement. Experiments demonstrate that GeoProg3D significantly outperforms existing 3D language fields and vision-language models across multiple tasks. To our knowledge, GeoProg3D is the first framework enabling compositional geographic reasoning in high-fidelity city-scale 3D environments via natural language. The code is available at https://snskysk.github.io/GeoProg3D/.