PRISE: Demystifying Deep Lucas-Kanade with Strongly Star-Convex Constraints for Multimodel Image Alignment

TL;DR

The paper introduces PRISE, a novel deep learning approach that incorporates strongly star-convex constraints into the Lucas-Kanade algorithm, significantly improving multimodel image alignment accuracy especially in challenging scenarios.

Contribution

It proposes a new deep star-convexification method for Lucas-Kanade, enabling better convergence and accuracy in multimodel image alignment tasks.

Findings

Achieves state-of-the-art results on benchmark datasets.

Significantly improves alignment accuracy for large distortions.

Provides an efficient sampling algorithm for training.

Abstract

The Lucas-Kanade (LK) method is a classic iterative homography estimation algorithm for image alignment, but often suffers from poor local optimality especially when image pairs have large distortions. To address this challenge, in this paper we propose a novel Deep Star-Convexified Lucas-Kanade (PRISE) method for multimodel image alignment by introducing strongly star-convex constraints into the optimization problem. Our basic idea is to enforce the neural network to approximately learn a star-convex loss landscape around the ground truth give any data to facilitate the convergence of the LK method to the ground truth through the high dimensional space defined by the network. This leads to a minimax learning problem, with contrastive (hinge) losses due to the definition of strong star-convexity that are appended to the original loss for training. We also provide an efficient sampling based algorithm to leverage the training cost, as well as some analysis on the quality of the solutions from PRISE. We further evaluate our approach on benchmark datasets such as MSCOCO, GoogleEarth, and GoogleMap, and demonstrate state-of-the-art results, especially for small pixel errors. Code can be downloaded from https://github.com/Zhang-VISLab.