Integrated Benchmarking and Design for Reproducible and Accessible Evaluation of Robotic Agents

TL;DR

This paper introduces a comprehensive framework for reproducible robotics research that integrates development and benchmarking, exemplified by the DUCKIENet system and the Duckietown Autolab, ensuring low variance and high reproducibility.

Contribution

It presents a novel integrated approach combining development and benchmarking in robotics, with a standardized, accessible infrastructure to enhance reproducibility.

Findings

Low variance in experimental results across different hardware and labs

Validated the repeatability of experiments using the infrastructure

Demonstrated the system's effectiveness in ensuring reproducibility

Abstract

As robotics matures and increases in complexity, it is more necessary than ever that robot autonomy research be reproducible. Compared to other sciences, there are specific challenges to benchmarking autonomy, such as the complexity of the software stacks, the variability of the hardware and the reliance on data-driven techniques, amongst others. In this paper, we describe a new concept for reproducible robotics research that integrates development and benchmarking, so that reproducibility is obtained "by design" from the beginning of the research/development processes. We first provide the overall conceptual objectives to achieve this goal and then a concrete instance that we have built: the DUCKIENet. One of the central components of this setup is the Duckietown Autolab, a remotely accessible standardized setup that is itself also relatively low-cost and reproducible. When evaluating agents, careful definition of interfaces allows users to choose among local versus remote evaluation using simulation, logs, or remote automated hardware setups. We validate the system by analyzing the repeatability of experiments conducted using the infrastructure and show that there is low variance across different robot hardware and across different remote labs.