General Principles of Learning-Based Multi-Agent Systems

TL;DR

This paper introduces a mathematical framework for designing large decentralized multi-agent systems using reinforcement learning, focusing on automatically setting reward functions to achieve global goals without conflicts.

Contribution

It presents a novel framework called COIN for creating multi-agent systems that self-organize to optimize collective objectives, demonstrated through two computer experiments.

Findings

COINs perform near optimally in complex problems

COINs avoid the tragedy of the commons

COINs achieve optimal performance in leader-follower scenarios

Abstract

We consider the problem of how to design large decentralized multi-agent systems (MAS's) in an automated fashion, with little or no hand-tuning. Our approach has each agent run a reinforcement learning algorithm. This converts the problem into one of how to automatically set/update the reward functions for each of the agents so that the global goal is achieved. In particular we do not want the agents to ``work at cross-purposes'' as far as the global goal is concerned. We use the term artificial COllective INtelligence (COIN) to refer to systems that embody solutions to this problem. In this paper we present a summary of a mathematical framework for COINs. We then investigate the real-world applicability of the core concepts of that framework via two computer experiments: we show that our COINs perform near optimally in a difficult variant of Arthur's bar problem (and in particular avoid the tragedy of the commons for that problem), and we also illustrate optimal performance for our COINs in the leader-follower problem.