NeSS-ST: Detecting Good and Stable Keypoints with a Neural Stability Score and the Shi-Tomasi Detector

TL;DR

NeSS-ST combines a traditional Shi-Tomasi detector with a neural network trained on a novel stability score to identify high-quality, stable keypoints without requiring pre-labeled datasets, achieving state-of-the-art results.

Contribution

It introduces a neural stability score and a training method that leverages the Shi-Tomasi detector to learn stable keypoints without dataset pre-labeling.

Findings

Achieves state-of-the-art performance on multiple benchmarks

Demonstrates strong generalization across diverse datasets

Does not require pre-labeled training data

Abstract

Learning a feature point detector presents a challenge both due to the ambiguity of the definition of a keypoint and, correspondingly, the need for specially prepared ground truth labels for such points. In our work, we address both of these issues by utilizing a combination of a hand-crafted Shi-Tomasi detector, a specially designed metric that assesses the quality of keypoints, the stability score (SS), and a neural network. We build on the principled and localized keypoints provided by the Shi-Tomasi detector and learn the neural network to select good feature points via the stability score. The neural network incorporates the knowledge from the training targets in the form of the neural stability score (NeSS). Therefore, our method is named NeSS-ST since it combines the Shi-Tomasi detector and the properties of the neural stability score. It only requires sets of images for training without dataset pre-labeling or the need for reconstructed correspondence labels. We evaluate NeSS-ST on HPatches, ScanNet, MegaDepth and IMC-PT demonstrating state-of-the-art performance and good generalization on downstream tasks.