Optimizing SLAM Evaluation Footprint Through Dynamic Range Coverage Analysis of Datasets

TL;DR

This paper analyzes the dynamic range coverage of SLAM datasets, identifies redundancy, and proposes a dynamic programming method to select a minimal subset of sequences that preserves coverage, thereby reducing evaluation effort.

Contribution

It introduces a systematic dataset characterization and a DP-based selection algorithm to optimize SLAM evaluation by reducing redundancy while maintaining coverage.

Findings

Redundancy exists within and between SLAM datasets.

The DP algorithm effectively reduces dataset size without losing coverage.

Evaluation effort can be minimized while preserving robustness assessment.

Abstract

Simultaneous Localization and Mapping (SLAM) is considered an ever-evolving problem due to its usage in many applications. Evaluation of SLAM is done typically using publicly available datasets which are increasing in number and the level of difficulty. Each dataset provides a certain level of dynamic range coverage that is a key aspect of measuring the robustness and resilience of SLAM. In this paper, we provide a systematic analysis of the dynamic range coverage of datasets based on a number of characterization metrics, and our analysis shows a huge level of redundancy within and between datasets. Subsequently, we propose a dynamic programming (DP) algorithm for eliminating the redundancy in the evaluation process of SLAM by selecting a subset of sequences that matches a single or multiple dynamic range coverage objectives. It is shown that, with the help of dataset characterization and DP selection algorithm, a reduction in the evaluation effort can be achieved while maintaining the same level of coverage. We also study how the evaluation process of a real-world SLAM system can be optimized utilizing the method proposed.