Be your own Benchmark: No-Reference Trajectory Metric on Registered Point Clouds

TL;DR

This paper introduces MOM, a no-reference trajectory quality metric for registered point clouds that correlates well with full-reference metrics, enabling benchmarking without ground truth in 3D mapping tasks.

Contribution

The paper proposes MOM, a novel no-reference trajectory metric based on orthogonal surface points, with a mathematical foundation and validated performance in synthetic and real datasets.

Findings

MOM strongly correlates with Relative Pose Error in experiments.

The algorithm effectively extracts orthogonal surface points for MOM.

Code is publicly available as a pip-package.

Abstract

This paper addresses the problem of assessing trajectory quality in conditions when no ground truth poses are available or when their accuracy is not enough for the specific task - for example, small-scale mapping in outdoor scenes. In our work, we propose a no-reference metric, Mutually Orthogonal Metric (MOM), that estimates the quality of the map from registered point clouds via the trajectory poses. MOM strongly correlates with full-reference trajectory metric Relative Pose Error, making it a trajectory benchmarking tool on setups where 3D sensing technologies are employed. We provide a mathematical foundation for such correlation and confirm it statistically in synthetic environments. Furthermore, since our metric uses a subset of points from mutually orthogonal surfaces, we provide an algorithm for the extraction of such subset and evaluate its performance in synthetic CARLA environment and on KITTI dataset. The code of the proposed metric is publicly available as pip-package.