HSFusion: A high-level vision task-driven infrared and visible image fusion network via semantic and geometric domain transformation

TL;DR

HSFusion is a novel infrared and visible image fusion network that uses semantic and geometric domain transformation via CycleGAN to better integrate high-level vision task information, improving fusion quality and segmentation performance.

Contribution

The paper introduces a domain transformation approach using CycleGAN to bridge semantic and geometric gaps, enhancing high-level vision task-driven image fusion.

Findings

Outperforms eleven state-of-the-art methods in visual quality

Enhances semantic segmentation accuracy

Effectively integrates semantic and geometric information

Abstract

Infrared and visible image fusion has been developed from vision perception oriented fusion methods to strategies which both consider the vision perception and high-level vision task. However, the existing task-driven methods fail to address the domain gap between semantic and geometric representation. To overcome these issues, we propose a high-level vision task-driven infrared and visible image fusion network via semantic and geometric domain transformation, terms as HSFusion. Specifically, to minimize the gap between semantic and geometric representation, we design two separate domain transformation branches by CycleGAN framework, and each includes two processes: the forward segmentation process and the reverse reconstruction process. CycleGAN is capable of learning domain transformation patterns, and the reconstruction process of CycleGAN is conducted under the constraint of these patterns. Thus, our method can significantly facilitate the integration of semantic and geometric information and further reduces the domain gap. In fusion stage, we integrate the infrared and visible features that extracted from the reconstruction process of two seperate CycleGANs to obtain the fused result. These features, containing varying proportions of semantic and geometric information, can significantly enhance the high level vision tasks. Additionally, we generate masks based on segmentation results to guide the fusion task. These masks can provide semantic priors, and we design adaptive weights for two distinct areas in the masks to facilitate image fusion. Finally, we conducted comparative experiments between our method and eleven other state-of-the-art methods, demonstrating that our approach surpasses others in both visual appeal and semantic segmentation task.