MapEval: Towards Unified, Robust and Efficient SLAM Map Evaluation Framework

TL;DR

MapEval introduces a unified, robust, and efficient framework for evaluating large-scale point cloud maps in SLAM, featuring novel metrics that improve accuracy and speed, and is openly available for community use.

Contribution

The paper presents MapEval, a new open-source framework with novel metrics based on Wasserstein distance for comprehensive, fast, and robust map quality assessment in SLAM applications.

Findings

Achieves 100-500 times faster evaluation than existing methods.

Provides consistent assessment guidelines for map quality.

Demonstrates robustness against noise in real-world datasets.

Abstract

Evaluating massive-scale point cloud maps in Simultaneous Localization and Mapping (SLAM) remains challenging, primarily due to the absence of unified, robust and efficient evaluation frameworks. We present MapEval, an open-source framework for comprehensive quality assessment of point cloud maps, specifically addressing SLAM scenarios where ground truth map is inherently sparse compared to the mapped environment. Through systematic analysis of existing evaluation metrics in SLAM applications, we identify their fundamental limitations and establish clear guidelines for consistent map quality assessment. Building upon these insights, we propose a novel Gaussian-approximated Wasserstein distance in voxelized space, enabling two complementary metrics under the same error standard: Voxelized Average Wasserstein Distance (AWD) for global geometric accuracy and Spatial Consistency Score (SCS) for local consistency evaluation. This theoretical foundation leads to significant improvements in both robustness against noise and computational efficiency compared to conventional metrics. Extensive experiments on both simulated and real-world datasets demonstrate that MapEval achieves at least \SI{100}{}-\SI{500}{} times faster while maintaining evaluation integrity. The MapEval library\footnote{\texttt{https://github.com/JokerJohn/Cloud\_Map\_Evaluation}} will be publicly available to promote standardized map evaluation practices in the robotics community.