Customizable Perturbation Synthesis for Robust SLAM Benchmarking

TL;DR

This paper introduces a customizable pipeline for synthesizing noisy data to evaluate the robustness of multi-modal SLAM models against diverse perturbations, addressing a gap in challenging simulation environments.

Contribution

It presents a novel perturbation synthesis toolbox and benchmark for assessing SLAM robustness, enabling controlled, diverse perturbations in simulation environments.

Findings

Existing SLAM models are vulnerable to real-world disturbances.

The proposed benchmark reveals specific susceptibilities of current SLAM systems.

The toolbox facilitates the creation of challenging noisy environments for robustness testing.

Abstract

Robustness is a crucial factor for the successful deployment of robots in unstructured environments, particularly in the domain of Simultaneous Localization and Mapping (SLAM). Simulation-based benchmarks have emerged as a highly scalable approach for robustness evaluation compared to real-world data collection. However, crafting a challenging and controllable noisy world with diverse perturbations remains relatively under-explored. To this end, we propose a novel, customizable pipeline for noisy data synthesis, aimed at assessing the resilience of multi-modal SLAM models against various perturbations. This pipeline incorporates customizable hardware setups, software components, and perturbed environments. In particular, we introduce comprehensive perturbation taxonomy along with a perturbation composition toolbox, allowing the transformation of clean simulations into challenging noisy environments. Utilizing the pipeline, we instantiate the Robust-SLAM benchmark, which includes diverse perturbation types, to evaluate the risk tolerance of existing advanced multi-modal SLAM models. Our extensive analysis uncovers the susceptibilities of existing SLAM models to real-world disturbance, despite their demonstrated accuracy in standard benchmarks. Our perturbation synthesis toolbox, SLAM robustness evaluation pipeline, and Robust-SLAM benchmark will be made publicly available at https://github.com/Xiaohao-Xu/SLAM-under-Perturbation/.