Learning and Crafting for the Wide Multiple Baseline Stereo

TL;DR

This thesis advances wide multiple baseline stereo (WxBS) by developing new descriptors, learning methods, and algorithms that improve matching across diverse imaging conditions, and introduces a comprehensive benchmark dataset.

Contribution

It introduces the WxBS problem, a new dataset, a novel descriptor training loss, a geometric feature learning method, an improved correspondence strategy, and the MODS matching algorithm.

Findings

HardNet descriptor achieves state-of-the-art performance.

MODS algorithm surpasses previous methods in large viewpoint changes.

A comprehensive benchmark for local features and robust estimation is provided.

Abstract

This thesis introduces the wide multiple baseline stereo (WxBS) problem. WxBS, a generalization of the standard wide baseline stereo problem, considers the matching of images that simultaneously differ in more than one image acquisition factor such as viewpoint, illumination, sensor type, or where object appearance changes significantly, e.g., over time. A new dataset with the ground truth, evaluation metric and baselines has been introduced. The thesis presents the following improvements of the WxBS pipeline. (i) A loss function, called HardNeg, for learning a local image descriptor that relies on hard negative mining within a mini-batch and on the maximization of the distance between the closest positive and the closest negative patches. (ii) The descriptor trained with the HardNeg loss, called HardNet, is compact and shows state-of-the-art performance in standard matching, patch verification and retrieval benchmarks. (iii) A method for learning the affine shape, orientation, and potentially other parameters related to geometric and appearance properties of local features. (iv) A tentative correspondences generation strategy which generalizes the standard first to second closest distance ratio is presented. The selection strategy, which shows performance superior to the standard method, is applicable to either hard-engineered descriptors like SIFT, LIOP, and MROGH or deeply learned like HardNet. (v) A feedback loop is introduced for the two-view matching problem, resulting in MODS -- matching with on-demand view synthesis -- algorithm. MODS is an algorithm that handles a viewing angle difference even larger than the previous state-of-the-art ASIFT algorithm, without a significant increase of computational cost over "standard" wide and narrow baseline approaches. Last, but not least, a comprehensive benchmark for local features and robust estimation algorithms is introduced.