Breaking Modality Disparity: Harmonized Representation for Infrared and Visible Image Registration

TL;DR

This paper introduces a novel scene-adaptive registration method for infrared and visible images that uses an invertible translation to create a modality-invariant domain, improving accuracy and reducing manual calibration efforts.

Contribution

It proposes a new invertible translation process for modality-invariant representation and a hierarchical framework for deformation rectification in infrared-visible image registration.

Findings

Outperforms state-of-the-art registration methods in experiments.

Introduces the first ground truth dataset for infrared-visible image alignment.

Demonstrates robustness in both synthetic and real-world scenarios.

Abstract

Since the differences in viewing range, resolution and relative position, the multi-modality sensing module composed of infrared and visible cameras needs to be registered so as to have more accurate scene perception. In practice, manual calibration-based registration is the most widely used process, and it is regularly calibrated to maintain accuracy, which is time-consuming and labor-intensive. To cope with these problems, we propose a scene-adaptive infrared and visible image registration. Specifically, in regard of the discrepancy between multi-modality images, an invertible translation process is developed to establish a modality-invariant domain, which comprehensively embraces the feature intensity and distribution of both infrared and visible modalities. We employ homography to simulate the deformation between different planes and develop a hierarchical framework to rectify the deformation inferred from the proposed latent representation in a coarse-to-fine manner. For that, the advanced perception ability coupled with the residual estimation conducive to the regression of sparse offsets, and the alternate correlation search facilitates a more accurate correspondence matching. Moreover, we propose the first ground truth available misaligned infrared and visible image dataset, involving three synthetic sets and one real-world set. Extensive experiments validate the effectiveness of the proposed method against the state-of-the-arts, advancing the subsequent applications.