Mediated Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a mediator-based approach to multi-agent reinforcement learning, aiming to establish socially beneficial equilibria that respect agent identities and boundaries, thus promoting cooperation without exploitation.

Contribution

It proposes a novel mediator framework inspired by mechanism design, trained with policy gradients to foster cooperation and social welfare in multi-agent settings.

Findings

Mediators can effectively promote cooperation in matrix and iterative games.

The approach achieves stable equilibria that respect agent boundaries.

Experimental results demonstrate the potential of mediators in MARL.

Abstract

The majority of Multi-Agent Reinforcement Learning (MARL) literature equates the cooperation of self-interested agents in mixed environments to the problem of social welfare maximization, allowing agents to arbitrarily share rewards and private information. This results in agents that forgo their individual goals in favour of social good, which can potentially be exploited by selfish defectors. We argue that cooperation also requires agents' identities and boundaries to be respected by making sure that the emergent behaviour is an equilibrium, i.e., a convention that no agent can deviate from and receive higher individual payoffs. Inspired by advances in mechanism design, we propose to solve the problem of cooperation, defined as finding socially beneficial equilibrium, by using mediators. A mediator is a benevolent entity that may act on behalf of agents, but only for the agents that agree to it. We show how a mediator can be trained alongside agents with policy gradient to maximize social welfare subject to constraints that encourage agents to cooperate through the mediator. Our experiments in matrix and iterative games highlight the potential power of applying mediators in MARL.