Deep Lucas-Kanade Homography for Multimodal Image Alignment

TL;DR

This paper introduces a deep learning-enhanced Lucas-Kanade method for accurately aligning multimodal images with large appearance differences by using learned feature maps that are invariant to appearance changes.

Contribution

It extends the traditional Lucas-Kanade algorithm with a novel deep feature map (DLKFM) that improves multimodal image alignment under appearance variations.

Findings

DLKFM recognizes invariant features across modalities.

The method achieves brightness consistency during alignment.

The Lucas-Kanade objective landscape becomes smoother, aiding convergence.

Abstract

Estimating homography to align image pairs captured by different sensors or image pairs with large appearance changes is an important and general challenge for many computer vision applications. In contrast to others, we propose a generic solution to pixel-wise align multimodal image pairs by extending the traditional Lucas-Kanade algorithm with networks. The key contribution in our method is how we construct feature maps, named as deep Lucas-Kanade feature map (DLKFM). The learned DLKFM can spontaneously recognize invariant features under various appearance-changing conditions. It also has two nice properties for the Lucas-Kanade algorithm: (1) The template feature map keeps brightness consistency with the input feature map, thus the color difference is very small while they are well-aligned. (2) The Lucas-Kanade objective function built on DLKFM has a smooth landscape around ground truth homography parameters, so the iterative solution of the Lucas-Kanade can easily converge to the ground truth. With those properties, directly updating the Lucas-Kanade algorithm on our feature maps will precisely align image pairs with large appearance changes. We share the datasets, code, and demo video online.