BusReF: Infrared-Visible images registration and fusion focus on reconstructible area using one set of features

TL;DR

BusReF presents a unified framework for infrared-visible image registration and fusion that enhances robustness and accuracy by integrating registration and fusion stages with a novel training strategy and a gradient-aware network.

Contribution

The paper introduces a combined registration and fusion framework for multi-modal images, utilizing a new training strategy and a gradient-aware network to improve performance.

Findings

Robust registration and fusion in a single framework.

Improved accuracy using mask-based loss functions.

Enhanced preservation of complementary information.

Abstract

In a scenario where multi-modal cameras are operating together, the problem of working with non-aligned images cannot be avoided. Yet, existing image fusion algorithms rely heavily on strictly registered input image pairs to produce more precise fusion results, as a way to improve the performance of downstream high-level vision tasks. In order to relax this assumption, one can attempt to register images first. However, the existing methods for registering multiple modalities have limitations, such as complex structures and reliance on significant semantic information. This paper aims to address the problem of image registration and fusion in a single framework, called BusRef. We focus on Infrared-Visible image registration and fusion task (IVRF). In this framework, the input unaligned image pairs will pass through three stages: Coarse registration, Fine registration and Fusion. It will be shown that the unified approach enables more robust IVRF. We also propose a novel training and evaluation strategy, involving the use of masks to reduce the influence of non-reconstructible regions on the loss functions, which greatly improves the accuracy and robustness of the fusion task. Last but not least, a gradient-aware fusion network is designed to preserve the complementary information. The advanced performance of this algorithm is demonstrated by