When Image Decomposition Meets Deep Learning: A Novel Infrared and Visible Image Fusion Method

TL;DR

This paper introduces a dual-stream auto-encoder network for infrared and visible image fusion, effectively decomposing and merging features to produce detailed, target-highlighted fused images superior to existing methods.

Contribution

A novel auto-encoder based fusion network with specialized loss functions and attention modules for improved infrared and visible image fusion.

Findings

Produces fused images with rich detail and highlighted targets.

Outperforms state-of-the-art fusion methods in qualitative and quantitative tests.

Maintains strong reproducibility across different images.

Abstract

Infrared and visible image fusion, as a hot topic in image processing and image enhancement, aims to produce fused images retaining the detail texture information in visible images and the thermal radiation information in infrared images. A critical step for this issue is to decompose features in different scales and to merge them separately. In this paper, we propose a novel dual-stream auto-encoder (AE) based fusion network. The core idea is that the encoder decomposes an image into base and detail feature maps with low- and high-frequency information, respectively, and that the decoder is responsible for the original image reconstruction. To this end, a well-designed loss function is established to make the base/detail feature maps similar/dissimilar. In the test phase, base and detail feature maps are respectively merged via an additional fusion layer, which contains a saliency weighted-based spatial attention module and a channel attention module to adaptively preserve more information from source images and to highlight the objects. Then the fused image is recovered by the decoder. Qualitative and quantitative results demonstrate that our method can generate fusion images containing highlighted targets and abundant detail texture information with strong reproducibility and meanwhile is superior to the state-of-the-art (SOTA) approaches.