Knowledge-aware Text-Image Retrieval for Remote Sensing Images

TL;DR

This paper introduces KTIR, a knowledge-aware method that enhances remote sensing text-image retrieval by leveraging external knowledge graphs, improving matching accuracy over existing methods.

Contribution

The paper proposes a novel knowledge-aware retrieval approach that enriches text queries and adapts models specifically for remote sensing images, addressing cross-modal information gaps.

Findings

Outperforms state-of-the-art retrieval methods on benchmark datasets.

Enriches text queries with external knowledge for better matching.

Improves domain adaptation of vision-language models in remote sensing.

Abstract

Image-based retrieval in large Earth observation archives is challenging because one needs to navigate across thousands of candidate matches only with the query image as a guide. By using text as information supporting the visual query, the retrieval system gains in usability, but at the same time faces difficulties due to the diversity of visual signals that cannot be summarized by a short caption only. For this reason, as a matching-based task, cross-modal text-image retrieval often suffers from information asymmetry between texts and images. To address this challenge, we propose a Knowledge-aware Text-Image Retrieval (KTIR) method for remote sensing images. By mining relevant information from an external knowledge graph, KTIR enriches the text scope available in the search query and alleviates the information gaps between texts and images for better matching. Moreover, by integrating domain-specific knowledge, KTIR also enhances the adaptation of pre-trained vision-language models to remote sensing applications. Experimental results on three commonly used remote sensing text-image retrieval benchmarks show that the proposed knowledge-aware method leads to varied and consistent retrievals, outperforming state-of-the-art retrieval methods.