Measuring robustness of Visual SLAM

TL;DR

This paper evaluates the robustness of RGB-D SLAM methods, particularly ORBSLAM2, through extensive experiments on existing datasets and a newly created large, realistic dataset, revealing robustness issues despite high accuracy.

Contribution

It introduces a large, diverse, and realistic HomeRobot dataset for evaluating RGB-D SLAM robustness and provides a comprehensive analysis of SLAM performance across various trajectories.

Findings

SLAM accuracy is often high, but robustness remains problematic.

Correlations exist between trajectory attributes and SLAM metrics.

The new dataset significantly expands the scope of SLAM evaluation.

Abstract

Simultaneous localization and mapping (SLAM) is an essential component of robotic systems. In this work we perform a feasibility study of RGB-D SLAM for the task of indoor robot navigation. Recent visual SLAM methods, e.g. ORBSLAM2 \cite{mur2017orb}, demonstrate really impressive accuracy, but the experiments in the papers are usually conducted on just a few sequences, that makes it difficult to reason about the robustness of the methods. Another problem is that all available RGB-D datasets contain the trajectories with very complex camera motions. In this work we extensively evaluate ORBSLAM2 to better understand the state-of-the-art. First, we conduct experiments on the popular publicly available datasets for RGB-D SLAM across the conventional metrics. We perform statistical analysis of the results and find correlations between the metrics and the attributes of the trajectories. Then, we introduce a new large and diverse HomeRobot dataset where we model the motions of a simple home robot. Our dataset is created using physically-based rendering with realistic lighting and contains the scenes composed by human designers. It includes thousands of sequences, that is two orders of magnitude greater than in previous works. We find that while in many cases the accuracy of SLAM is very good, the robustness is still an issue.