EarthVL: A Progressive Earth Vision-Language Understanding and Generation Framework

TL;DR

EarthVL introduces a comprehensive framework combining a new dataset and a multi-stage model for advanced Earth scene understanding and generation, focusing on geospatial object reasoning and city planning applications.

Contribution

The paper presents EarthVLNet and EarthVLSet, a novel dataset and progressive model for Earth vision-language tasks, emphasizing object-relational reasoning and scene comprehension.

Findings

EarthVLNet outperforms existing methods on semantic segmentation and VQA benchmarks.

Segmentation features improve VQA performance across datasets.

Open-ended VQA tasks require more advanced vision and language models.

Abstract

Earth vision has achieved milestones in geospatial object recognition but lacks exploration in object-relational reasoning, limiting comprehensive scene understanding. To address this, a progressive Earth vision-language understanding and generation framework is proposed, including a multi-task dataset (EarthVLSet) and a semantic-guided network (EarthVLNet). Focusing on city planning applications, EarthVLSet includes 10.9k sub-meter resolution remote sensing images, land-cover masks, and 761.5k textual pairs involving both multiple-choice and open-ended visual question answering (VQA) tasks. In an object-centric way, EarthVLNet is proposed to progressively achieve semantic segmentation, relational reasoning, and comprehensive understanding. The first stage involves land-cover segmentation to generate object semantics for VQA guidance. Guided by pixel-wise semantics, the object awareness based large language model (LLM) performs relational reasoning and knowledge summarization to generate the required answers. As for optimization, the numerical difference loss is proposed to dynamically add difference penalties, addressing the various objects' statistics. Three benchmarks, including semantic segmentation, multiple-choice, and open-ended VQA demonstrated the superiorities of EarthVLNet, yielding three future directions: 1) segmentation features consistently enhance VQA performance even in cross-dataset scenarios; 2) multiple-choice tasks show greater sensitivity to the vision encoder than to the language decoder; and 3) open-ended tasks necessitate advanced vision encoders and language decoders for an optimal performance. We believe this dataset and method will provide a beneficial benchmark that connects ''image-mask-text'', advancing geographical applications for Earth vision.