Crowd against the machine: A simulation-based benchmark tool to evaluate and compare robot capabilities to navigate a human crowd

TL;DR

This paper introduces a simulation-based benchmark tool designed to evaluate and compare robot navigation capabilities in crowded human environments, aiming to standardize testing and improve safety and efficiency.

Contribution

It develops an initial simulation architecture with scenarios and metrics for benchmarking robot crowd navigation, advancing the evaluation methods in this domain.

Findings

Early results show the benchmark tool's relevance and potential for standardization.

The simulation environment effectively models crowded scenarios for robot testing.

The proposed metrics can assess safety and efficiency in robot navigation.

Abstract

The evaluation of robot capabilities to navigate human crowds is essential to conceive new robots intended to operate in public spaces. This paper initiates the development of a benchmark tool to evaluate such capabilities; our long term vision is to provide the community with a simulation tool that generates virtual crowded environment to test robots, to establish standard scenarios and metrics to evaluate navigation techniques in terms of safety and efficiency, and thus, to install new methods to benchmarking robots' crowd navigation capabilities. This paper presents the architecture of the simulation tools, introduces first scenarios and evaluation metrics, as well as early results to demonstrate that our solution is relevant to be used as a benchmark tool.