Reinforced Feature Points: Optimizing Feature Detection and Description for a High-Level Task

TL;DR

This paper introduces a novel end-to-end training approach for feature detectors that integrates the entire vision pipeline, improving high-level task performance by using reinforcement learning to optimize detection and description.

Contribution

It presents a new training methodology that embeds feature detection in a complete pipeline, enhancing high-level task accuracy beyond traditional low-level matching optimization.

Findings

Enhanced accuracy in relative pose estimation.

Effective end-to-end training for any architecture predicting key points.

Improved performance over traditional handcrafted features.

Abstract

We address a core problem of computer vision: Detection and description of 2D feature points for image matching. For a long time, hand-crafted designs, like the seminal SIFT algorithm, were unsurpassed in accuracy and efficiency. Recently, learned feature detectors emerged that implement detection and description using neural networks. Training these networks usually resorts to optimizing low-level matching scores, often pre-defining sets of image patches which should or should not match, or which should or should not contain key points. Unfortunately, increased accuracy for these low-level matching scores does not necessarily translate to better performance in high-level vision tasks. We propose a new training methodology which embeds the feature detector in a complete vision pipeline, and where the learnable parameters are trained in an end-to-end fashion. We overcome the discrete nature of key point selection and descriptor matching using principles from reinforcement learning. As an example, we address the task of relative pose estimation between a pair of images. We demonstrate that the accuracy of a state-of-the-art learning-based feature detector can be increased when trained for the task it is supposed to solve at test time. Our training methodology poses little restrictions on the task to learn, and works for any architecture which predicts key point heat maps, and descriptors for key point locations.