Emergent Coordination Through Competition
Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess,, Thore Graepel

TL;DR
This paper investigates how cooperative behaviors emerge among reinforcement learning agents in a complex multi-agent soccer environment, highlighting the role of competitive training and evaluation methods.
Contribution
It introduces a new multi-agent soccer environment and demonstrates how decentralized population-based training fosters the emergence of cooperation.
Findings
Agents progress from random to cooperative behaviors.
Shaping rewards can influence long-term team cooperation.
Evaluation scheme based on game theory assesses agent performance without predefined tasks.
Abstract
We study the emergence of cooperative behaviors in reinforcement learning agents by introducing a challenging competitive multi-agent soccer environment with continuous simulated physics. We demonstrate that decentralized, population-based training with co-play can lead to a progression in agents' behaviors: from random, to simple ball chasing, and finally showing evidence of cooperation. Our study highlights several of the challenges encountered in large scale multi-agent training in continuous control. In particular, we demonstrate that the automatic optimization of simple shaping rewards, not themselves conducive to co-operative behavior, can lead to long-horizon team behavior. We further apply an evaluation scheme, grounded by game theoretic principals, that can assess agent performance in the absence of pre-defined evaluation tasks or human baselines.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSports Analytics and Performance · Reinforcement Learning in Robotics
