SSPFusion: A Semantic Structure-Preserving Approach for Infrared and Visible Image Fusion

TL;DR

SSPFusion introduces a novel semantic structure-preserving method for multi-modality image fusion, effectively maintaining structural consistency and outperforming existing methods in infrared-visible and medical image fusion tasks.

Contribution

The paper proposes a new fusion approach with a structural feature extractor and a multi-scale SPF module guided by Sobel-based structural signals, ensuring semantic structure preservation.

Findings

Outperforms nine state-of-the-art methods in qualitative evaluations.

Demonstrates strong generalization in infrared-visible and medical image fusion.

Produces high-quality, structure-preserving fused images.

Abstract

Most existing learning-based multi-modality image fusion (MMIF) methods suffer from significant structure inconsistency due to their inappropriate usage of structural features at the semantic level. To alleviate these issues, we propose a semantic structure-preserving fusion approach for MMIF, namely SSPFusion. At first, we design a structural feature extractor (SFE) to extract the prominent structural features from multiple input images. Concurrently, we introduce a transformation function with Sobel operator to generate self-supervised structural signals in these extracted features. Subsequently, we design a multi-scale structure-preserving fusion (SPF) module, guided by the generated structural signals, to merge the structural features of input images. This process ensures the preservation of semantic structure consistency between the resultant fusion image and the input images. Through the synergy of these two robust modules of SFE and SPF, our method can generate high-quality fusion images and demonstrate good generalization ability. Experimental results, on both infrared-visible image fusion and medical image fusion tasks, demonstrate that our method outperforms nine state-of-the-art methods in terms of both qualitative and quantitative evaluations. The code is publicly available at https://github.com/QiaoYang-CV/SSPFUSION.