GeoVision Labeler: Zero-Shot Geospatial Classification with Vision and Language Models

TL;DR

GeoVision Labeler (GVL) is a novel zero-shot geospatial image classification framework that uses vision and language models to generate human-readable descriptions and classify images without task-specific training, achieving high accuracy on benchmark datasets.

Contribution

GVL introduces a modular, interpretable pipeline combining vision and language models for flexible zero-shot geospatial classification, outperforming existing methods without task-specific pretraining.

Findings

Achieves up to 93.2% accuracy on SpaceNet v7 binary task.

Effective hierarchical classification with recursive LLM-driven clustering.

Competitive performance on multi-class geospatial datasets.

Abstract

Classifying geospatial imagery remains a major bottleneck for applications such as disaster response and land-use monitoring-particularly in regions where annotated data is scarce or unavailable. Existing tools (e.g., RS-CLIP) that claim zero-shot classification capabilities for satellite imagery nonetheless rely on task-specific pretraining and adaptation to reach competitive performance. We introduce GeoVision Labeler (GVL), a strictly zero-shot classification framework: a vision Large Language Model (vLLM) generates rich, human-readable image descriptions, which are then mapped to user-defined classes by a conventional Large Language Model (LLM). This modular, and interpretable pipeline enables flexible image classification for a large range of use cases. We evaluated GVL across three benchmarks-SpaceNet v7, UC Merced, and RESISC45. It achieves up to 93.2% zero-shot accuracy on the binary Buildings vs. No Buildings task on SpaceNet v7. For complex multi-class classification tasks (UC Merced, RESISC45), we implemented a recursive LLM-driven clustering to form meta-classes at successive depths, followed by hierarchical classification-first resolving coarse groups, then finer distinctions-to deliver competitive zero-shot performance. GVL is open-sourced at https://github.com/microsoft/geo-vision-labeler to catalyze adoption in real-world geospatial workflows.