Emergent Coordination Through Competition

TL;DR

This paper investigates how cooperative behaviors emerge among reinforcement learning agents in a complex multi-agent soccer environment, highlighting the role of competitive training and evaluation methods.

Contribution

It introduces a new multi-agent soccer environment and demonstrates how decentralized population-based training fosters the emergence of cooperation.

Findings

Agents progress from random to cooperative behaviors.

Shaping rewards can influence long-term team cooperation.

Evaluation scheme based on game theory assesses agent performance without predefined tasks.

Abstract

We study the emergence of cooperative behaviors in reinforcement learning agents by introducing a challenging competitive multi-agent soccer environment with continuous simulated physics. We demonstrate that decentralized, population-based training with co-play can lead to a progression in agents' behaviors: from random, to simple ball chasing, and finally showing evidence of cooperation. Our study highlights several of the challenges encountered in large scale multi-agent training in continuous control. In particular, we demonstrate that the automatic optimization of simple shaping rewards, not themselves conducive to co-operative behavior, can lead to long-horizon team behavior. We further apply an evaluation scheme, grounded by game theoretic principals, that can assess agent performance in the absence of pre-defined evaluation tasks or human baselines.