Towards Using Multiple Iterated, Reproduced, and Replicated Experiments with Robots (MIRRER) for Evaluation and Benchmarking

TL;DR

This paper introduces MIRRER, a framework for evaluating and benchmarking robotics experiments through multiple, reproduced, and replicated tests to improve reproducibility and generalizability in robotics research.

Contribution

It proposes a novel conceptual framework that unites evaluation, benchmarking, and replication for robotics experiments, addressing current gaps in reproducibility and comparability.

Findings

Initial framework MIRRER outlined for robotics evaluation.

Highlights open issues in applying the framework.

Aims to standardize reproducibility and benchmarking in robotics.

Abstract

The robotics research field lacks formalized definitions and frameworks for evaluating advanced capabilities including generalizability (the ability for robots to perform tasks under varied contexts) and reproducibility (the performance of a reproduced robot capability in different labs under the same experimental conditions). This paper presents an initial conceptual framework, MIRRER, that unites the concepts of performance evaluation, benchmarking, and reproduced/replicated experimentation in order to facilitate comparable robotics research. Several open issues with the application of the framework are also presented.