l0-Regularized Sparse Coding-based Interpretable Network for Multi-Modal Image Fusion

TL;DR

This paper introduces FNet, an interpretable neural network for multi-modal image fusion that uses an $ ext{l}_0$-regularized sparse coding approach to effectively combine features from different sensor images, improving visualization and detection tasks.

Contribution

The paper proposes a novel $ ext{l}_0$-regularized multi-modal convolutional sparse coding model and an interpretable deep unfolding network for multi-modal image fusion, with an inverse fusion model for training.

Findings

FNet achieves high-quality fusion across eight datasets.

FNet improves object detection and semantic segmentation.

The network's intermediate results are visually interpretable.

Abstract

Multi-modal image fusion (MMIF) enhances the information content of the fused image by combining the unique as well as common features obtained from different modality sensor images, improving visualization, object detection, and many more tasks. In this work, we introduce an interpretable network for the MMIF task, named FNet, based on an $\ell_0$-regularized multi-modal convolutional sparse coding (MCSC) model. Specifically, for solving the $\ell_0$-regularized CSC problem, we design a learnable $\ell_0$-regularized sparse coding (LZSC) block in a principled manner through deep unfolding. Given different modality source images, FNet first separates the unique and common features from them using the LZSC block and then these features are combined to generate the final fused image. Additionally, we propose an $\ell_0$-regularized MCSC model for the inverse fusion process. Based on this model, we introduce an interpretable inverse fusion network named IFNet, which is utilized during FNet's training. Extensive experiments show that FNet achieves high-quality fusion results across eight different MMIF datasets. Furthermore, we show that FNet enhances downstream object detection \textcolor[rgb]{ 0, 0, 0}{and semantic segmentation} in visible-thermal image pairs. We have also visualized the intermediate results of FNet, which demonstrates the good interpretability of our network. Link for code and models: https://github.com/gargi884/FNet-MMIF.