Efficient Neighbourhood Consensus Networks via Submanifold Sparse Convolutions

TL;DR

This paper introduces Sparse-NCNet, a memory-efficient and faster neighbourhood consensus network using submanifold sparse convolutions, achieving state-of-the-art local correspondence accuracy in image matching tasks.

Contribution

It proposes a novel sparse correlation tensor processing method with submanifold convolutions, significantly reducing memory and computation while improving localization accuracy.

Findings

Reduces memory and inference time by over 10x with equivalent accuracy.

Achieves state-of-the-art results on HPatches and InLoc benchmarks.

Improves localization accuracy through higher resolution processing and a new relocalization module.

Abstract

In this work we target the problem of estimating accurately localised correspondences between a pair of images. We adopt the recent Neighbourhood Consensus Networks that have demonstrated promising performance for difficult correspondence problems and propose modifications to overcome their main limitations: large memory consumption, large inference time and poorly localised correspondences. Our proposed modifications can reduce the memory footprint and execution time more than $10\times$, with equivalent results. This is achieved by sparsifying the correlation tensor containing tentative matches, and its subsequent processing with a 4D CNN using submanifold sparse convolutions. Localisation accuracy is significantly improved by processing the input images in higher resolution, which is possible due to the reduced memory footprint, and by a novel two-stage correspondence relocalisation module. The proposed Sparse-NCNet method obtains state-of-the-art results on the HPatches Sequences and InLoc visual localisation benchmarks, and competitive results in the Aachen Day-Night benchmark.