rSoccer: A Framework for Studying Reinforcement Learning in Small and Very Small Size Robot Soccer

TL;DR

This paper introduces rSoccer, an open-source simulator and framework for applying reinforcement learning to small and very small size robot soccer, facilitating research and competition in this domain.

Contribution

It provides a specialized simulator and a set of benchmark environments for reinforcement learning in robot soccer, enabling easier experimentation and evaluation.

Findings

Demonstrated learning capabilities of two state-of-the-art RL methods

Identified limitations of RL methods in certain scenarios

Facilitated research in end-to-end RL for robot soccer

Abstract

Reinforcement learning is an active research area with a vast number of applications in robotics, and the RoboCup competition is an interesting environment for studying and evaluating reinforcement learning methods. A known difficulty in applying reinforcement learning to robotics is the high number of experience samples required, being the use of simulated environments for training the agents followed by transfer learning to real-world (sim-to-real) a viable path. This article introduces an open-source simulator for the IEEE Very Small Size Soccer and the Small Size League optimized for reinforcement learning experiments. We also propose a framework for creating OpenAI Gym environments with a set of benchmarks tasks for evaluating single-agent and multi-agent robot soccer skills. We then demonstrate the learning capabilities of two state-of-the-art reinforcement learning methods as well as their limitations in certain scenarios introduced in this framework. We believe this will make it easier for more teams to compete in these categories using end-to-end reinforcement learning approaches and further develop this research area.