The Effect of Ground Truth Accuracy on the Evaluation of Localization Systems

TL;DR

This paper presents a theoretical framework and algorithms to analyze and correct the impact of ground truth errors on the evaluation of localization systems, improving accuracy in performance assessment.

Contribution

It introduces a novel theoretical approach and algorithms to account for ground truth inaccuracies, enhancing the reliability of localization system evaluations.

Findings

Algorithms accurately estimate real errors within 4% of actual CDF

Map error correction improves median error estimate by 150%

Framework effectively corrects localization error assessments in real environments

Abstract

The ability to accurately evaluate the performance of location determination systems is crucial for many applications. Typically, the performance of such systems is obtained by comparing ground truth locations with estimated locations. However, these ground truth locations are usually obtained by clicking on a map or using other worldwide available technologies like GPS. This introduces ground truth errors that are due to the marking process, map distortions, or inherent GPS inaccuracy. In this paper, we present a theoretical framework for analyzing the effect of ground truth errors on the evaluation of localization systems. Based on that, we design two algorithms for computing the real algorithmic error from the validation error and marking/map ground truth errors, respectively. We further establish bounds on different performance metrics. Validation of our theoretical assumptions and analysis using real data collected in a typical environment shows the ability of our theoretical framework to correct the estimated error of a localization algorithm in the presence of ground truth errors. Specifically, our marking error algorithm matches the real error CDF within 4%, and our map error algorithm provides a more accurate estimate of the median/tail error by 150%/72% when the map is shifted by 6m.