Learning to Align Images using Weak Geometric Supervision

TL;DR

This paper introduces a weakly supervised method for image alignment that learns local descriptors and estimates homography simultaneously without labeled data, enabling cross-modality alignment and competitive performance.

Contribution

A novel joint learning approach for image alignment and descriptor training from scratch using weak supervision and iterative homography updates.

Findings

Successfully aligned RGB and NIR images without labeled data

Achieved competitive accuracy on HPatches benchmark

Enabled training of generalizable descriptors from automatically aligned images

Abstract

Image alignment tasks require accurate pixel correspondences, which are usually recovered by matching local feature descriptors. Such descriptors are often derived using supervised learning on existing datasets with ground truth correspondences. However, the cost of creating such datasets is usually prohibitive. In this paper, we propose a new approach to align two images related by an unknown 2D homography where the local descriptor is learned from scratch from the images and the homography is estimated simultaneously. Our key insight is that a siamese convolutional neural network can be trained jointly while iteratively updating the homography parameters by optimizing a single loss function. Our method is currently weakly supervised because the input images need to be roughly aligned. We have used this method to align images of different modalities such as RGB and near-infra-red (NIR) without using any prior labeled data. Images automatically aligned by our method were then used to train descriptors that generalize to new images. We also evaluated our method on RGB images. On the HPatches benchmark, our method achieves comparable accuracy to deep local descriptors that were trained offline in a supervised setting.