UnsuperPoint: End-to-end Unsupervised Interest Point Detector and Descriptor

TL;DR

UnsuperPoint is an unsupervised, real-time interest point detector and descriptor trained with a novel self-supervised approach, achieving high performance without requiring ground truth annotations or multiple training rounds.

Contribution

It introduces a fully end-to-end trainable unsupervised interest point detector and descriptor using a novel loss function and regression approach, eliminating the need for pseudo ground truth.

Findings

Runs at 323 fps at 224x320 resolution

Achieves comparable or better performance than state-of-the-art methods

Effective in repeatability, localization, matching, and homography estimation

Abstract

It is hard to create consistent ground truth data for interest points in natural images, since interest points are hard to define clearly and consistently for a human annotator. This makes interest point detectors non-trivial to build. In this work, we introduce an unsupervised deep learning-based interest point detector and descriptor. Using a self-supervised approach, we utilize a siamese network and a novel loss function that enables interest point scores and positions to be learned automatically. The resulting interest point detector and descriptor is UnsuperPoint. We use regression of point positions to 1) make UnsuperPoint end-to-end trainable and 2) to incorporate non-maximum suppression in the model. Unlike most trainable detectors, it requires no generation of pseudo ground truth points, no structure-from-motion-generated representations and the model is learned from only one round of training. Furthermore, we introduce a novel loss function to regularize network predictions to be uniformly distributed. UnsuperPoint runs in real-time with 323 frames per second (fps) at a resolution of $224\times320$ and 90 fps at $480\times640$. It is comparable or better than state-of-the-art performance when measured for speed, repeatability, localization, matching score and homography estimation on the HPatch dataset.