JSSFF: A Joint Structural-Semantic Fusion Framework for Remote Sensing Image Captioning

TL;DR

This paper introduces JSSFF, a novel framework for remote sensing image captioning that fuses structural and semantic features to improve caption accuracy, especially in complex images.

Contribution

It proposes an edge-aware fusion method incorporating original and edge-aware images into the encoder, enhancing feature representation and boundary detection.

Findings

Outperforms baseline models in quantitative metrics

Generates more accurate and relevant captions in complex scenarios

Demonstrates effectiveness of edge-aware fusion in remote sensing images

Abstract

The encoder-decoder framework has become widely popular nowadays. In this model, the encoder extracts informative visual features from an input image, and the decoder employs a sequence-to-sequence formulation to generate the corresponding textual description from these features. The existing models focus more on the decision part. However, extracting meaningful information from the image can help the decoder generate an accurate caption by providing information about the objects and their relationship. Remote sensing images are highly complex. One major challenge is detecting objects that extend beyond their visible boundaries due to occlusion, overlapping structures, and unclear edges. Hence, there is a need to design an approach that can effectively capture both high-level semantics and low-level spatial details for accurate caption generation. In this work, we have proposed an edge-aware fusion method by incorporating the original image and its edge-aware version into the encoder to enhance feature representation and boundary awareness. We used a comparison-based beam search (CBBS) to generate captions to achieve a balanced trade-off between quantitative metrics and qualitative caption relevance through fairness-based comparison of candidate captions. Experimental results demonstrate our model's superiority over several baseline models in quantitative and qualitative perspectives.