Infrared and Visible Image Fusion with ResNet and zero-phase component analysis

TL;DR

This paper introduces a novel infrared and visible image fusion method using ResNet for feature extraction and zero-phase component analysis for normalization, resulting in improved fusion quality over existing methods.

Contribution

The paper proposes a new fusion framework combining ResNet and ZCA for better feature processing and fusion performance, which is not addressed by prior deep learning methods.

Findings

Achieves superior objective assessment scores

Enhances visual quality of fused images

Outperforms existing fusion methods in experiments

Abstract

Feature extraction and processing tasks play a key role in Image Fusion, and the fusion performance is directly affected by the different features and processing methods undertaken. By contrast, most of deep learning-based methods use deep features directly without feature extraction or processing. This leads to the fusion performance degradation in some cases. To solve these drawbacks, we propose a deep features and zero-phase component analysis (ZCA) based novel fusion framework is this paper. Firstly, the residual network (ResNet) is used to extract deep features from source images. Then ZCA is utilized to normalize the deep features and obtain initial weight maps. The final weight maps are obtained by employing a soft-max operation in association with the initial weight maps. Finally, the fused image is reconstructed using a weighted-averaging strategy. Compared with the existing fusion methods, experimental results demonstrate that the proposed framework achieves better performance in both objective assessment and visual quality. The code of our fusion algorithm is available at https://github.com/hli1221/imagefusion_resnet50