From Perfect to Noisy World Simulation: Customizable Embodied Multi-modal Perturbations for SLAM Robustness Benchmarking

TL;DR

This paper introduces a customizable pipeline for synthesizing diverse sensor and motion perturbations in simulation environments to evaluate the robustness of multi-modal SLAM models, revealing their vulnerabilities to various disturbances.

Contribution

It presents a novel, comprehensive pipeline and toolbox for generating challenging noisy environments for SLAM benchmarking, along with the large-scale Noisy-Replica benchmark for robustness assessment.

Findings

Neural SLAM models are susceptible to perturbations despite high accuracy in standard benchmarks.

Non-neural SLAM models also show vulnerabilities under noisy conditions.

The pipeline enables systematic evaluation of SLAM robustness across diverse perturbations.

Abstract

Embodied agents require robust navigation systems to operate in unstructured environments, making the robustness of Simultaneous Localization and Mapping (SLAM) models critical to embodied agent autonomy. While real-world datasets are invaluable, simulation-based benchmarks offer a scalable approach for robustness evaluations. However, the creation of a challenging and controllable noisy world with diverse perturbations remains under-explored. To this end, we propose a novel, customizable pipeline for noisy data synthesis, aimed at assessing the resilience of multi-modal SLAM models against various perturbations. The pipeline comprises a comprehensive taxonomy of sensor and motion perturbations for embodied multi-modal (specifically RGB-D) sensing, categorized by their sources and propagation order, allowing for procedural composition. We also provide a toolbox for synthesizing these perturbations, enabling the transformation of clean environments into challenging noisy simulations. Utilizing the pipeline, we instantiate the large-scale Noisy-Replica benchmark, which includes diverse perturbation types, to evaluate the risk tolerance of existing advanced RGB-D SLAM models. Our extensive analysis uncovers the susceptibilities of both neural (NeRF and Gaussian Splatting -based) and non-neural SLAM models to disturbances, despite their demonstrated accuracy in standard benchmarks. Our code is publicly available at https://github.com/Xiaohao-Xu/SLAM-under-Perturbation.