Bridging the Gap between Multi-focus and Multi-modal: A Focused Integration Framework for Multi-modal Image Fusion

TL;DR

This paper introduces a novel multi-modal image fusion framework that effectively integrates multi-focus visible images and infrared images by decomposing and fusing structural and texture components, improving visual perception and task performance.

Contribution

The paper proposes a semi-sparsity-based decomposition and a multi-scale fusion operator tailored for multi-focus and multi-modal images, addressing a key challenge in MMIF applications.

Findings

Outperforms state-of-the-art methods in visual perception.

Enhances object detection and depth estimation accuracy.

Demonstrates robustness across multiple datasets.

Abstract

Multi-modal image fusion (MMIF) integrates valuable information from different modality images into a fused one. However, the fusion of multiple visible images with different focal regions and infrared images is a unprecedented challenge in real MMIF applications. This is because of the limited depth of the focus of visible optical lenses, which impedes the simultaneous capture of the focal information within the same scene. To address this issue, in this paper, we propose a MMIF framework for joint focused integration and modalities information extraction. Specifically, a semi-sparsity-based smoothing filter is introduced to decompose the images into structure and texture components. Subsequently, a novel multi-scale operator is proposed to fuse the texture components, capable of detecting significant information by considering the pixel focus attributes and relevant data from various modal images. Additionally, to achieve an effective capture of scene luminance and reasonable contrast maintenance, we consider the distribution of energy information in the structural components in terms of multi-directional frequency variance and information entropy. Extensive experiments on existing MMIF datasets, as well as the object detection and depth estimation tasks, consistently demonstrate that the proposed algorithm can surpass the state-of-the-art methods in visual perception and quantitative evaluation. The code is available at https://github.com/ixilai/MFIF-MMIF.