Are We Ready for Robust and Resilient SLAM? A Framework For Quantitative Characterization of SLAM Datasets

TL;DR

This paper introduces a new framework for quantitatively characterizing SLAM datasets to better evaluate their impact on SLAM system robustness and resilience, addressing a gap in current research.

Contribution

The paper proposes a novel, extendable framework for the quantitative analysis and comparison of SLAM datasets, linking dataset characteristics to SLAM robustness and resilience.

Findings

Characterization of KITTI, EuroC-MAV, and TUM-VI datasets using the framework

Insights into how dataset conditions affect SLAM performance

Establishment of a foundation for dataset-driven robustness evaluation

Abstract

Reliability of SLAM systems is considered one of the critical requirements in modern autonomous systems. This directed the efforts to developing many state-of-the-art systems, creating challenging datasets, and introducing rigorous metrics to measure SLAM performance. However, the link between datasets and performance in the robustness/resilience context has rarely been explored. In order to fill this void, characterization of the operating conditions of SLAM systems is essential in order to provide an environment for quantitative measurement of robustness and resilience. In this paper, we argue that for proper evaluation of SLAM performance, the characterization of SLAM datasets serves as a critical first step. The study starts by reviewing previous efforts for quantitative characterization of SLAM datasets. Then, the problem of perturbation characterization is discussed and the linkage to SLAM robustness/resilience is established. After that, we propose a novel, generic and extendable framework for quantitative analysis and comparison of SLAM datasets. Additionally, a description of different characterization parameters is provided. Finally, we demonstrate the application of our framework by presenting the characterization results of three SLAM datasets: KITTI, EuroC-MAV, and TUM-VI highlighting the level of insights achieved by the proposed framework.