A Semantic-Aware and Multi-Guided Network for Infrared-Visible Image Fusion

TL;DR

This paper introduces a novel multi-branch network that models inter-feature relationships and high-level graph reasoning to improve infrared-visible image fusion, enhancing downstream tasks like object detection and segmentation.

Contribution

It proposes a three-branch encoder-decoder architecture with graph reasoning and cross attention modules for better multi-modality feature fusion.

Findings

Outperforms state-of-the-art in image fusion tasks.

Achieves higher mAP and mIoU in object detection and segmentation.

Effectively preserves high-frequency details and modality-shared information.

Abstract

Multi-modality image fusion aims at fusing modality-specific (complementarity) and modality-shared (correlation) information from multiple source images. To tackle the problem of the neglect of inter-feature relationships, high-frequency information loss, and the limited attention to downstream tasks, this paper focuses on how to model correlation-driven decomposing features and reason high-level graph representation by efficiently extracting complementary information and aggregating multi-guided features. We propose a three-branch encoder-decoder architecture along with corresponding fusion layers as the fusion strategy. Firstly, shallow features from individual modalities are extracted by a depthwise convolution layer combined with the transformer block. In the three parallel branches of the encoder, Cross Attention and Invertible Block (CAI) extracts local features and preserves high-frequency texture details. Base Feature Extraction Module (BFE) captures long-range dependencies and enhances modality-shared information. Graph Reasoning Module (GR) is introduced to reason high-level cross-modality relations and simultaneously extract low-level detail features as CAI's modality-specific complementary information. Experiments demonstrate the competitive results compared with state-of-the-art methods in visible/infrared image fusion and medical image fusion tasks. Moreover, the proposed algorithm surpasses the state-of-the-art methods in terms of subsequent tasks, averagely scoring 8.27% mAP@0.5 higher in object detection and 5.85% mIoU higher in semantic segmentation. The code is avaliable at https://github.com/Abraham-Einstein/SMFNet/.