TSJNet: A Multi-modality Target and Semantic Awareness Joint-driven Image Fusion Network

TL;DR

TSJNet is a novel multi-modality image fusion network that leverages semantic information and attention mechanisms to improve the fusion of multimodal images for enhanced semantic segmentation and object detection.

Contribution

The paper introduces TSJNet, a fusion network that jointly utilizes high-level semantic information and multi-scale features to enhance multimodal image fusion performance.

Findings

Achieves state-of-the-art results on five public datasets.

Improves mAP@0.5 by 7.97% and mIoU by 10.88% over existing methods.

Demonstrates strong generalization with a new UAV dataset.

Abstract

This study aims to address the problem of incomplete information in unimodal images for semantic segmentation and object detection tasks. Existing multimodal fusion methods suffer from limited capability in discriminative modeling of multi-scale semantic structures and salient target regions, which further restricts the effective fusion of task-related semantic details and target information across modalities. To tackle these challenges, this paper proposes a novel fusion network termed TSJNet, which leverages the semantic information output by high-level tasks in a joint manner to guide the fusion process. Specifically, we design a multi-dimensional feature extraction module with dual parallel branches to capture multi-scale and salient features. Meanwhile, a data-agnostic spatial attention module embedded in the decoder dynamically calibrates attention allocation across different data domains, significantly enhancing the model's generalization ability. To optimize both fusion and advanced visual tasks, we balance performance by combining fusion loss with semantic losses. Additionally, we have developed a multimodal unmanned aerial vehicle (UAV) dataset covering multiple scenarios (UMS). Extensive experiments demonstrate that TSJNet achieves outstanding performance on five public datasets (MSRS, M\textsuperscript{3}FD, RoadScene, LLVIP, and TNO) and our UMS dataset. The generated fusion results exhibit favorable visual effects, and compared to state-of-the-art methods, the mean average precision (mAP@0.5) and mean intersection over union (mIoU) for object detection and segmentation, respectively, improve by 7.97\% and 10.88\%.The code and the dataset has been publicly released at https://github.com/XylonXu01/TSJNet.