S4Fusion: Saliency-aware Selective State Space Model for Infrared Visible Image Fusion

TL;DR

S4Fusion introduces a saliency-aware model that effectively combines infrared and visible images by capturing global spatial information and adaptively emphasizing salient targets, leading to improved fusion quality.

Contribution

The paper presents S4Fusion, a novel fusion model that incorporates a cross-modal spatial awareness module and uncertainty minimization to enhance salient target preservation.

Findings

Produces higher quality fused images

Improves downstream task performance

Effectively captures global spatial information

Abstract

As one of the tasks in Image Fusion, Infrared and Visible Image Fusion aims to integrate complementary information captured by sensors of different modalities into a single image. The Selective State Space Model (SSSM), known for its ability to capture long-range dependencies, has demonstrated its potential in the field of computer vision. However, in image fusion, current methods underestimate the potential of SSSM in capturing the global spatial information of both modalities. This limitation prevents the simultaneous consideration of the global spatial information from both modalities during interaction, leading to a lack of comprehensive perception of salient targets. Consequently, the fusion results tend to bias towards one modality instead of adaptively preserving salient targets. To address this issue, we propose the Saliency-aware Selective State Space Fusion Model (S4Fusion). In our S4Fusion, the designed Cross-Modal Spatial Awareness Module (CMSA) can simultaneously focus on global spatial information from both modalities while facilitating their interaction, thereby comprehensively capturing complementary information. Additionally, S4Fusion leverages a pre-trained network to perceive uncertainty in the fused images. By minimizing this uncertainty, S4Fusion adaptively highlights salient targets from both images. Extensive experiments demonstrate that our approach produces high-quality images and enhances performance in downstream tasks.