Learning multiagent coordination in the absence of communication channels

TL;DR

This paper introduces a reinforcement learning approach for multiagent coordination in a communication-free iterated prisoner's dilemma tournament, demonstrating emergent strategies and consistent behaviors that enable agents to collude and win.

Contribution

It develops a Q-learning based framework for multiagent coordination without communication, analyzing emergent behaviors and strategies in a discrete game setting.

Findings

Agents learn effective collusion strategies.

Emergent behaviors include benevolent dictators.

Agents develop consistent action symbology for collusion.

Abstract

In this work, we develop a reinforcement learning protocol for a multiagent coordination task in a discrete state and action space: an iterated prisoner's dilemma game extended into a team based, winner-take all tournament, which forces the agents to collude in order to maximize their reward. By disallowing extra communication channels, the agents are forced to embed their coordination strategy into their actions in the prisoner's dilemma game. We develop a representation of the iterated prisoners dilemma that makes it amenable to Q-learning. We find that the reinforcement learning strategy is able to consistently train agents that can win the winner take all iterated prisoners dilemma tournament. By using a game with discrete state and action space, we are able to better analyze and understand both the dynamics and the communication protocols that are established between the agents. We find that the agents adapt a number of interesting behaviors, such as the formation of benevolent dictators, that minimize inequality of scores. We also find that the agents settle on a remarkably consistent symbology in their actions, such that agents from independent trials are able to collude with each other without further training.