On the Limits of Pseudo Ground Truth in Visual Camera Re-localisation

TL;DR

This paper critically examines how the choice of pseudo ground truth algorithms influences the evaluation of visual camera re-localisation methods, revealing biases and questioning established performance rankings.

Contribution

It analyzes the impact of reference algorithms on re-localisation benchmarks and highlights potential biases affecting method rankings.

Findings

Evaluation outcomes vary with the reference algorithm used.

Claims about the superiority of learning-based over classical methods are challenged.

The influence of the reference algorithm's similarity to the evaluated methods is significant.

Abstract

Benchmark datasets that measure camera pose accuracy have driven progress in visual re-localisation research. To obtain poses for thousands of images, it is common to use a reference algorithm to generate pseudo ground truth. Popular choices include Structure-from-Motion (SfM) and Simultaneous-Localisation-and-Mapping (SLAM) using additional sensors like depth cameras if available. Re-localisation benchmarks thus measure how well each method replicates the results of the reference algorithm. This begs the question whether the choice of the reference algorithm favours a certain family of re-localisation methods. This paper analyzes two widely used re-localisation datasets and shows that evaluation outcomes indeed vary with the choice of the reference algorithm. We thus question common beliefs in the re-localisation literature, namely that learning-based scene coordinate regression outperforms classical feature-based methods, and that RGB-D-based methods outperform RGB-based methods. We argue that any claims on ranking re-localisation methods should take the type of the reference algorithm, and the similarity of the methods to the reference algorithm, into account.