MARLadona -- Towards Cooperative Team Play Using Multi-Agent Reinforcement Learning

TL;DR

This paper presents MARLadona, a decentralized multi-agent reinforcement learning framework for cooperative robot soccer, achieving high win rates and providing insights into agent behavior and intentions.

Contribution

Introduces MARLadona, a novel MARL training pipeline and open-source environment for cooperative robot soccer, surpassing heuristic strategies in performance.

Findings

Achieves 66.8% win rate against state-of-the-art heuristic agent.

Provides interpretability of agent policies and intentions.

Develops an open-source multi-agent soccer environment.

Abstract

Robot soccer, in its full complexity, poses an unsolved research challenge. Current solutions heavily rely on engineered heuristic strategies, which lack robustness and adaptability. Deep reinforcement learning has gained significant traction in various complex robotics tasks such as locomotion, manipulation, and competitive games (e.g., AlphaZero, OpenAI Five), making it a promising solution to the robot soccer problem. This paper introduces MARLadona. A decentralized multi-agent reinforcement learning (MARL) training pipeline capable of producing agents with sophisticated team play behavior, bridging the shortcomings of heuristic methods. Furthermore, we created an open-source multi-agent soccer environment. Utilizing our MARL framework and a modified global entity encoder (GEE) as our core architecture, our approach achieves a 66.8% win rate against HELIOS agent, which employs a state-of-the-art heuristic strategy. In addition, we provided an in-depth analysis of the policy behavior and interpreted the agent's intention using the critic network.