R2D2: Repeatable and Reliable Detector and Descriptor

TL;DR

This paper introduces R2D2, a neural network that jointly learns repeatable keypoints and reliable descriptors, improving performance in local feature detection and description for computer vision tasks.

Contribution

It proposes a novel joint learning approach with a predictor for descriptor discriminativeness, enhancing reliability and repeatability over existing methods.

Findings

Outperforms state-of-the-art detectors and descriptors on HPatches

Sets a new record on Aachen Day-Night localization dataset

Produces sparse, repeatable, and reliable keypoints

Abstract

Interest point detection and local feature description are fundamental steps in many computer vision applications. Classical methods for these tasks are based on a detect-then-describe paradigm where separate handcrafted methods are used to first identify repeatable keypoints and then represent them with a local descriptor. Neural networks trained with metric learning losses have recently caught up with these techniques, focusing on learning repeatable saliency maps for keypoint detection and learning descriptors at the detected keypoint locations. In this work, we argue that salient regions are not necessarily discriminative, and therefore can harm the performance of the description. Furthermore, we claim that descriptors should be learned only in regions for which matching can be performed with high confidence. We thus propose to jointly learn keypoint detection and description together with a predictor of the local descriptor discriminativeness. This allows us to avoid ambiguous areas and leads to reliable keypoint detections and descriptions. Our detection-and-description approach, trained with self-supervision, can simultaneously output sparse, repeatable and reliable keypoints that outperforms state-of-the-art detectors and descriptors on the HPatches dataset. It also establishes a record on the recently released Aachen Day-Night localization dataset.