Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval

TL;DR

This paper introduces a novel asymmetric multimodal feature matching network (AMFMN) for cross-modal remote sensing image retrieval, addressing multi-scale features and redundancy issues to improve accuracy on a newly constructed fine-grained dataset.

Contribution

The paper proposes a new AMFMN model with multi-scale visual self-attention and a dynamic triplet loss, along with a challenging fine-grained RSITMD dataset for improved retrieval performance.

Findings

Achieves state-of-the-art results on four RS text-image datasets.

Effectively filters redundant features and handles multi-scale visual information.

Demonstrates robustness on a newly constructed fine-grained dataset.

Abstract

Remote sensing (RS) cross-modal text-image retrieval has attracted extensive attention for its advantages of flexible input and efficient query. However, traditional methods ignore the characteristics of multi-scale and redundant targets in RS image, leading to the degradation of retrieval accuracy. To cope with the problem of multi-scale scarcity and target redundancy in RS multimodal retrieval task, we come up with a novel asymmetric multimodal feature matching network (AMFMN). Our model adapts to multi-scale feature inputs, favors multi-source retrieval methods, and can dynamically filter redundant features. AMFMN employs the multi-scale visual self-attention (MVSA) module to extract the salient features of RS image and utilizes visual features to guide the text representation. Furthermore, to alleviate the positive samples ambiguity caused by the strong intraclass similarity in RS image, we propose a triplet loss function with dynamic variable margin based on prior similarity of sample pairs. Finally, unlike the traditional RS image-text dataset with coarse text and higher intraclass similarity, we construct a fine-grained and more challenging Remote sensing Image-Text Match dataset (RSITMD), which supports RS image retrieval through keywords and sentence separately and jointly. Experiments on four RS text-image datasets demonstrate that the proposed model can achieve state-of-the-art performance in cross-modal RS text-image retrieval task.