SAM Guided Semantic and Motion Changed Region Mining for Remote Sensing Change Captioning

TL;DR

This paper introduces a novel change captioning method for remote sensing images that combines the Segment Anything Model with a knowledge graph and Transformer-based fusion to improve region awareness and temporal alignment, achieving state-of-the-art results.

Contribution

It proposes integrating SAM for region-level change detection and a knowledge graph for enhanced semantic understanding into remote sensing change captioning.

Findings

Achieves state-of-the-art performance on benchmark datasets

Effectively delineates semantic and motion change regions

Enhances captioning accuracy with region-aware features

Abstract

Remote sensing change captioning is an emerging and popular research task that aims to describe, in natural language, the content of interest that has changed between two remote sensing images captured at different times. Existing methods typically employ CNNs/Transformers to extract visual representations from the given images or incorporate auxiliary tasks to enhance the final results, with weak region awareness and limited temporal alignment. To address these issues, this paper explores the use of the SAM (Segment Anything Model) foundation model to extract region-level representations and inject region-of-interest knowledge into the captioning framework. Specifically, we employ a CNN/Transformer model to extract global-level vision features, leverage the SAM foundation model to delineate semantic- and motion-level change regions, and utilize a specially constructed knowledge graph to provide information about objects of interest. These heterogeneous sources of information are then fused via cross-attention, and a Transformer decoder is used to generate the final natural language description of the observed changes. Extensive experimental results demonstrate that our method achieves state-of-the-art performance across multiple widely used benchmark datasets. The source code of this paper will be released on https://github.com/Event-AHU/SAM_ChangeCaptioning