Simulation leagues: Enabling replicable and robust investigation of complex robotic systems

TL;DR

Simulation leagues like RoboCup provide a powerful, replicable platform for complex robotic research, enabling parallel experiments and robust evaluation of competition formats that are infeasible with physical robots.

Contribution

This paper reviews RoboCup simulation leagues and demonstrates their utility in evaluating competition formats, highlighting their role in robust, cost-effective robotics research.

Findings

Hybrid competition format reduces performance ranking fluctuations.

Parallel experiments enable statistically-significant analysis.

Simulation environments facilitate cost-effective, reproducible research.

Abstract

Physically-realistic simulated environments are powerful platforms for enabling measurable, replicable and statistically-robust investigation of complex robotic systems. Such environments are epitomised by the RoboCup simulation leagues, which have been successfully utilised to conduct massively-parallel experiments in topics including: optimisation of bipedal locomotion, self-localisation from noisy perception data and planning complex multi-agent strategies without direct agent-to-agent communication. Many of these systems are later transferred to physical robots, making the simulation leagues invaluable well-beyond the scope of simulated soccer matches. In this study, we provide an overview of the RoboCup simulation leagues and describe their properties as they pertain to replicable and robust robotics research. To demonstrate their utility directly, we leverage the ability to run parallelised experiments to evaluate different competition formats (e.g. round robin) for the RoboCup 2D simulation league. Our results demonstrate that a previously-proposed hybrid format minimises fluctuations from 'true' (statistically-significant) team performance rankings within the time constraints of the RoboCup world finals. Our experimental analysis would be impossible with physical robots alone, and we encourage other researchers to explore the potential for enriching their experimental pipelines with simulated components, both to minimise experimental costsand enable others to replicate and expand upon their results in a hardware-independent manner.