Bi-directional Self-Registration for Misaligned Infrared-Visible Image Fusion

TL;DR

This paper introduces a self-supervised bi-directional registration framework for aligning misaligned infrared-visible image pairs, improving multi-modal image fusion without requiring ground truth data.

Contribution

It proposes a novel self-supervised registration method using proxy data generators and a neighborhood dynamic alignment loss to handle modal gaps.

Findings

Effective in aligning misaligned multi-modal images

Improves image fusion quality

Outperforms existing registration methods

Abstract

Acquiring accurately aligned multi-modal image pairs is fundamental for achieving high-quality multi-modal image fusion. To address the lack of ground truth in current multi-modal image registration and fusion methods, we propose a novel self-supervised \textbf{B}i-directional \textbf{S}elf-\textbf{R}egistration framework (\textbf{B-SR}). Specifically, B-SR utilizes a proxy data generator (PDG) and an inverse proxy data generator (IPDG) to achieve self-supervised global-local registration. Visible-infrared image pairs with spatially misaligned differences are aligned to obtain global differences through the registration module. The same image pairs are processed by PDG, such as cropping, flipping, stitching, etc., and then aligned to obtain local differences. IPDG converts the obtained local differences into pseudo-global differences, which are used to perform global-local difference consistency with the global differences. Furthermore, aiming at eliminating the effect of modal gaps on the registration module, we design a neighborhood dynamic alignment loss to achieve cross-modal image edge alignment. Extensive experiments on misaligned multi-modal images demonstrate the effectiveness of the proposed method in multi-modal image alignment and fusion against the competing methods. Our code will be publicly available.