Knowledge-aware Visual Question Generation for Remote Sensing Images

TL;DR

This paper introduces KRSVQG, a knowledge-aware model for generating diverse and contextually rich questions from remote sensing images by integrating external knowledge and image captioning techniques.

Contribution

It presents a novel approach that incorporates external knowledge and image captioning to improve question diversity and contextual understanding in remote sensing images.

Findings

KRSVQG outperforms existing methods on NWPU-300 and TextRS-300 datasets.

Generated questions are more diverse and knowledge-enriched.

Model effectively grounds questions in both image content and external domain knowledge.

Abstract

With the rapid development of remote sensing image archives, asking questions about images has become an effective way of gathering specific information or performing image retrieval. However, automatically generated image-based questions tend to be simplistic and template-based, which hinders the real deployment of question answering or visual dialogue systems. To enrich and diversify the questions, we propose a knowledge-aware remote sensing visual question generation model, KRSVQG, that incorporates external knowledge related to the image content to improve the quality and contextual understanding of the generated questions. The model takes an image and a related knowledge triplet from external knowledge sources as inputs and leverages image captioning as an intermediary representation to enhance the image grounding of the generated questions. To assess the performance of KRSVQG, we utilized two datasets that we manually annotated: NWPU-300 and TextRS-300. Results on these two datasets demonstrate that KRSVQG outperforms existing methods and leads to knowledge-enriched questions, grounded in both image and domain knowledge.