SLAM Adversarial Lab: An Extensible Framework for Visual SLAM Robustness Evaluation under Adverse Conditions

TL;DR

SAL is a modular framework that systematically evaluates the robustness of visual SLAM systems under various adversarial weather conditions, enabling severity-level analysis and extensibility.

Contribution

We introduce SAL, an extensible framework that models adversarial conditions as perturbations, supports severity levels, and evaluates multiple SLAM algorithms under diverse adverse scenarios.

Findings

Seven SLAM algorithms tested across three datasets.

SAL effectively identifies failure points at different severity levels.

Framework supports easy addition of new perturbations and datasets.

Abstract

We present SAL (SLAM Adversarial Lab), a modular framework for evaluating visual SLAM systems under adversarial conditions such as fog and rain. SAL represents each adversarial condition as a perturbation that transforms an existing dataset into an adversarial dataset. When transforming a dataset, SAL supports severity levels using easily-interpretable real-world units such as meters for fog visibility. SAL's extensible architecture decouples datasets, perturbations, and SLAM algorithms through common interfaces, so users can add new components without rewriting integration code. Moreover, SAL includes a search procedure that finds the severity level of a perturbation at which a SLAM system fails. To showcase the capabilities of SAL, our evaluation integrates seven SLAM algorithms and evaluates them across three datasets under weather, camera, and video transport perturbations.