Learning to Play Soccer From Scratch: Sample-Efficient Emergent Coordination through Curriculum-Learning and Competition

TL;DR

This paper introduces a sample-efficient multi-agent reinforcement learning approach for 2v2 soccer, using curriculum learning, experience sharing, and frame-skipping to achieve high-quality play in under 40 million interactions.

Contribution

It presents a novel multi-stage curriculum learning framework with experience sharing and frame-skipping for efficient multi-agent policy learning in soccer.

Findings

High-quality soccer behaviors learned in under 40M interactions.

Effective multi-stage curriculum improves learning efficiency.

Experience sharing enhances training stability and performance.

Abstract

This work proposes a scheme that allows learning complex multi-agent behaviors in a sample efficient manner, applied to 2v2 soccer. The problem is formulated as a Markov game, and solved using deep reinforcement learning. We propose a basic multi-agent extension of TD3 for learning the policy of each player, in a decentralized manner. To ease learning, the task of 2v2 soccer is divided in three stages: 1v0, 1v1 and 2v2. The process of learning in multi-agent stages (1v1 and 2v2) uses agents trained on a previous stage as fixed opponents. In addition, we propose using experience sharing, a method that shares experience from a fixed opponent, trained in a previous stage, for training the agent currently learning, and a form of frame-skipping, to raise performance significantly. Our results show that high quality soccer play can be obtained with our approach in just under 40M interactions. A summarized video of the resulting game play can be found in https://youtu.be/f25l1j1U9RM.