RSVLM-QA: A Benchmark Dataset for Remote Sensing Vision Language Model-based Question Answering

TL;DR

RSVLM-QA is a comprehensive, large-scale dataset for remote sensing visual question answering, created using advanced LLMs and segmentation data, to improve model evaluation and development.

Contribution

The paper introduces RSVLM-QA, a novel large-scale remote sensing VQA dataset with rich annotations and diverse question types, generated via an innovative dual-track pipeline involving GPT-4.1.

Findings

RSVLM-QA contains 13,820 images and 162,373 VQA pairs.

Benchmarking shows current VLMs have room for improvement on RS VQA tasks.

The dataset enhances evaluation of reasoning capabilities in remote sensing models.

Abstract

Visual Question Answering (VQA) in remote sensing (RS) is pivotal for interpreting Earth observation data. However, existing RS VQA datasets are constrained by limitations in annotation richness, question diversity, and the assessment of specific reasoning capabilities. This paper introduces RSVLM-QA dataset, a new large-scale, content-rich VQA dataset for the RS domain. RSVLM-QA is constructed by integrating data from several prominent RS segmentation and detection datasets: WHU, LoveDA, INRIA, and iSAID. We employ an innovative dual-track annotation generation pipeline. Firstly, we leverage Large Language Models (LLMs), specifically GPT-4.1, with meticulously designed prompts to automatically generate a suite of detailed annotations including image captions, spatial relations, and semantic tags, alongside complex caption-based VQA pairs. Secondly, to address the challenging task of object counting in RS imagery, we have developed a specialized automated process that extracts object counts directly from the original segmentation data; GPT-4.1 then formulates natural language answers from these counts, which are paired with preset question templates to create counting QA pairs. RSVLM-QA comprises 13,820 images and 162,373 VQA pairs, featuring extensive annotations and diverse question types. We provide a detailed statistical analysis of the dataset and a comparison with existing RS VQA benchmarks, highlighting the superior depth and breadth of RSVLM-QA's annotations. Furthermore, we conduct benchmark experiments on Six mainstream Vision Language Models (VLMs), demonstrating that RSVLM-QA effectively evaluates and challenges the understanding and reasoning abilities of current VLMs in the RS domain. We believe RSVLM-QA will serve as a pivotal resource for the RS VQA and VLM research communities, poised to catalyze advancements in the field.