Questions beyond Pixels: Integrating Commonsense Knowledge in Visual Question Generation for Remote Sensing

TL;DR

This paper introduces KRSVQG, a knowledge-aware model for generating diverse, grounded questions about remote sensing images by integrating external commonsense knowledge and employing vision-language pre-training.

Contribution

The paper presents a novel KRSVQG model that incorporates external knowledge triplets and uses captioning as an intermediary, improving question diversity and grounding in remote sensing images.

Findings

KRSVQG outperforms existing methods in metrics and human assessments.

Constructed two new datasets: NWPU-300 and TextRS-300.

Generated questions are richer and better grounded in image and domain knowledge.

Abstract

With the rapid development of remote sensing image archives, asking questions about images has become an effective way of gathering specific information or performing semantic image retrieval. However, current automatically generated questions tend to be simplistic and template-based, which hinders the deployment of question answering or visual dialogue systems for real-world applications. To enrich and diversify the questions with both image content and commonsense knowledge, we propose a Knowledge-aware Remote Sensing Visual Question Generation model (KRSVQG). The proposed model incorporates related knowledge triplets from external knowledge sources to broaden the question content, while employing image captioning as an intermediary representation to ground questions to the corresponding images. Moreover, KRSVQG utilizes a vision-language pre-training and fine-tuning strategy, enabling the model's adaptation to low data regimes. To evaluate the proposed KRSVQG model, we construct two knowledge-aware remote sensing visual question generation datasets: the NWPU-300 dataset and the TextRS-300 dataset. Evaluations, including metrics and human assessment, demonstrate that KRSVQG outperforms existing methods and leads to rich questions, grounded in both image and domain knowledge. As a key practice in vision-language research, knowledge-aware visual question generation advances the understanding of image content beyond pixels, facilitating the development of knowledge-enriched vision-language systems with vision-grounded human commonsense.