Towards a multimodal framework for remote sensing image change retrieval and captioning

TL;DR

This paper introduces a novel multimodal framework for remote sensing image change retrieval and captioning, utilizing contrastive learning to enhance change detection and captioning capabilities on bi-temporal RS data.

Contribution

It presents a new foundation model that combines contrastive learning with captioning for bi-temporal remote sensing images, addressing the gap in multimodal RS applications.

Findings

Model achieves competitive captioning performance.

Enhances text-image retrieval in change detection.

Utilizes LEVIR-CC dataset for training and evaluation.

Abstract

Recently, there has been increasing interest in multimodal applications that integrate text with other modalities, such as images, audio and video, to facilitate natural language interactions with multimodal AI systems. While applications involving standard modalities have been extensively explored, there is still a lack of investigation into specific data modalities such as remote sensing (RS) data. Despite the numerous potential applications of RS data, including environmental protection, disaster monitoring and land planning, available solutions are predominantly focused on specific tasks like classification, captioning and retrieval. These solutions often overlook the unique characteristics of RS data, such as its capability to systematically provide information on the same geographical areas over time. This ability enables continuous monitoring of changes in the underlying landscape. To address this gap, we propose a novel foundation model for bi-temporal RS image pairs, in the context of change detection analysis, leveraging Contrastive Learning and the LEVIR-CC dataset for both captioning and text-image retrieval. By jointly training a contrastive encoder and captioning decoder, our model add text-image retrieval capabilities, in the context of bi-temporal change detection, while maintaining captioning performances that are comparable to the state of the art. We release the source code and pretrained weights at: https://github.com/rogerferrod/RSICRC.