PCNet: A Structure Similarity Enhancement Method for Multispectral and Multimodal Image Registration

TL;DR

PCNet is a lightweight neural network that enhances structure similarity in multispectral and multimodal images, significantly improving registration accuracy and outperforming state-of-the-art methods without extensive tuning.

Contribution

The paper introduces PCNet, a novel structure similarity enhancement network leveraging phase congruency, which is effective across various multispectral and multimodal image types with minimal parameters.

Findings

PCNet surpasses DHN in registration performance.

PCNet boosts the accuracy of deep-learning registration methods.

Over 89.9% of images achieve under 1 pixel ACE after PCNet processing.

Abstract

Multispectral and multimodal images are of important usage in the field of multi-source visual information fusion. Due to the alternation or movement of image devices, the acquired multispectral and multimodal images are usually misaligned, and hence image registration is pre-requisite. Different from the registration of common images, the registration of multispectral or multimodal images is a challenging problem due to the nonlinear variation of intensity and gradient. To cope with this challenge, we propose the phase congruency network (PCNet) to enhance the structure similarity of multispectral or multimodal images. The images can then be aligned using the similarity-enhanced feature maps produced by the network. PCNet is constructed under the inspiration of the well-known phase congruency. The network embeds the phase congruency prior into two simple trainable layers and series of modified learnable Gabor kernels. Thanks to the prior knowledge, once trained, PCNet is applicable on a variety of multispectral and multimodal data such as flash/no-flash and RGB/NIR images without additional further tuning. The prior also makes the network lightweight. The trainable parameters of PCNet are 2400 times less than the deep-learning registration method DHN, while its registration performance surpasses DHN. Experimental results validate that PCNet outperforms current state-of-the-art conventional multimodal registration algorithms. Besides, PCNet can act as a complementary part of the deep-learning registration methods, which significantly boosts their registration accuracy. The percentage of the number of images under 1 pixel average corner error (ACE) of UDHN is raised from 0.2% to 89.9% after the processing of PCNet.