Inducing Cooperative behaviour in Sequential-Social dilemmas through Multi-Agent Reinforcement Learning using Status-Quo Loss

TL;DR

This paper introduces a novel approach using Status-Quo Loss and GameDistill to promote cooperative behavior in multi-agent reinforcement learning within sequential social dilemmas, addressing the challenge of selfish convergence.

Contribution

It proposes SQLoss to encourage stability and cooperation, and GameDistill to extract policies from visual social dilemma games, advancing multi-agent cooperation methods.

Findings

Agents with SQLoss develop cooperative strategies.

GameDistill effectively extracts policies from visual inputs.

Combined approach leads to socially desirable behaviors.

Abstract

In social dilemma situations, individual rationality leads to sub-optimal group outcomes. Several human engagements can be modeled as a sequential (multi-step) social dilemmas. However, in contrast to humans, Deep Reinforcement Learning agents trained to optimize individual rewards in sequential social dilemmas converge to selfish, mutually harmful behavior. We introduce a status-quo loss (SQLoss) that encourages an agent to stick to the status quo, rather than repeatedly changing its policy. We show how agents trained with SQLoss evolve cooperative behavior in several social dilemma matrix games. To work with social dilemma games that have visual input, we propose GameDistill. GameDistill uses self-supervision and clustering to automatically extract cooperative and selfish policies from a social dilemma game. We combine GameDistill and SQLoss to show how agents evolve socially desirable cooperative behavior in the Coin Game.