Comparing Feature Detectors: A bias in the repeatability criteria, and how to correct it

TL;DR

This paper identifies a bias in the standard repeatability criterion used for comparing feature detectors in computer vision and proposes a correction method that considers descriptor overlap, leading to a revised evaluation of detector performance.

Contribution

The paper introduces a modified repeatability measure that accounts for descriptor overlap, providing a fairer comparison of diverse feature detectors.

Findings

The classic repeatability criterion is biased towards overlapping detections.

The proposed overlap-aware measure significantly alters the ranking of feature detectors.

Re-evaluation with the new measure shows different conclusions about detector performance.

Abstract

Most computer vision application rely on algorithms finding local correspondences between different images. These algorithms detect and compare stable local invariant descriptors centered at scale-invariant keypoints. Because of the importance of the problem, new keypoint detectors and descriptors are constantly being proposed, each one claiming to perform better (or to be complementary) to the preceding ones. This raises the question of a fair comparison between very diverse methods. This evaluation has been mainly based on a repeatability criterion of the keypoints under a series of image perturbations (blur, illumination, noise, rotations, homotheties, homographies, etc). In this paper, we argue that the classic repeatability criterion is biased towards algorithms producing redundant overlapped detections. To compensate this bias, we propose a variant of the repeatability rate taking into account the descriptors overlap. We apply this variant to revisit the popular benchmark by Mikolajczyk et al., on classic and new feature detectors. Experimental evidence shows that the hierarchy of these feature detectors is severely disrupted by the amended comparator.