Learning a Graph Neural Network with Cross Modality Interaction for Image Fusion

TL;DR

This paper introduces IGNet, a graph neural network architecture that enhances infrared and visible image fusion by facilitating cross-modality interaction, leading to improved image quality and downstream task performance.

Contribution

The paper proposes a novel GNN-based framework with cross-modality interaction and leader nodes for better feature fusion and semantic learning in multi-modality image fusion.

Findings

Outperforms state-of-the-art methods in detection and segmentation metrics.

Produces visually appealing fused images with richer details.

Demonstrates effectiveness across multiple datasets (TNO, MFNet, M3FD).

Abstract

Infrared and visible image fusion has gradually proved to be a vital fork in the field of multi-modality imaging technologies. In recent developments, researchers not only focus on the quality of fused images but also evaluate their performance in downstream tasks. Nevertheless, the majority of methods seldom put their eyes on the mutual learning from different modalities, resulting in fused images lacking significant details and textures. To overcome this issue, we propose an interactive graph neural network (GNN)-based architecture between cross modality for fusion, called IGNet. Specifically, we first apply a multi-scale extractor to achieve shallow features, which are employed as the necessary input to build graph structures. Then, the graph interaction module can construct the extracted intermediate features of the infrared/visible branch into graph structures. Meanwhile, the graph structures of two branches interact for cross-modality and semantic learning, so that fused images can maintain the important feature expressions and enhance the performance of downstream tasks. Besides, the proposed leader nodes can improve information propagation in the same modality. Finally, we merge all graph features to get the fusion result. Extensive experiments on different datasets (TNO, MFNet and M3FD) demonstrate that our IGNet can generate visually appealing fused images while scoring averagely 2.59% mAP@.5 and 7.77% mIoU higher in detection and segmentation than the compared state-of-the-art methods. The source code of the proposed IGNet can be available at https://github.com/lok-18/IGNet.