TSJNet: A Multi-modality Target and Semantic Awareness Joint-driven Image Fusion Network
Yuchan Jie, Yushen Xu, Xiaosong Li, Huafeng Li, Haishu Tan, Feiping Nie

TL;DR
TSJNet is a novel multi-modality image fusion network that leverages semantic information and attention mechanisms to improve the fusion of multimodal images for enhanced semantic segmentation and object detection.
Contribution
The paper introduces TSJNet, a fusion network that jointly utilizes high-level semantic information and multi-scale features to enhance multimodal image fusion performance.
Findings
Achieves state-of-the-art results on five public datasets.
Improves [email protected] by 7.97% and mIoU by 10.88% over existing methods.
Demonstrates strong generalization with a new UAV dataset.
Abstract
This study aims to address the problem of incomplete information in unimodal images for semantic segmentation and object detection tasks. Existing multimodal fusion methods suffer from limited capability in discriminative modeling of multi-scale semantic structures and salient target regions, which further restricts the effective fusion of task-related semantic details and target information across modalities. To tackle these challenges, this paper proposes a novel fusion network termed TSJNet, which leverages the semantic information output by high-level tasks in a joint manner to guide the fusion process. Specifically, we design a multi-dimensional feature extraction module with dual parallel branches to capture multi-scale and salient features. Meanwhile, a data-agnostic spatial attention module embedded in the decoder dynamically calibrates attention allocation across different data…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Image Fusion Techniques · Image Retrieval and Classification Techniques
MethodsFocus
