Learning Reciprocity in Complex Sequential Social Dilemmas

TL;DR

This paper introduces an online reinforcement learning algorithm that enables agents to exhibit reciprocal behavior in complex, extended social dilemmas, promoting cooperation in multi-agent settings.

Contribution

It presents a novel reinforcement learning method capable of learning reciprocity in temporally extended social dilemmas, extending beyond simple matrix games.

Findings

Reciprocal agents induce pro-social outcomes in multi-agent social dilemmas.

Agents' behavior is strongly influenced by their co-players.

The method works in both 2-player and 5-player social dilemmas.

Abstract

Reciprocity is an important feature of human social interaction and underpins our cooperative nature. What is more, simple forms of reciprocity have proved remarkably resilient in matrix game social dilemmas. Most famously, the tit-for-tat strategy performs very well in tournaments of Prisoner's Dilemma. Unfortunately this strategy is not readily applicable to the real world, in which options to cooperate or defect are temporally and spatially extended. Here, we present a general online reinforcement learning algorithm that displays reciprocal behavior towards its co-players. We show that it can induce pro-social outcomes for the wider group when learning alongside selfish agents, both in a $2$-player Markov game, and in $5$-player intertemporal social dilemmas. We analyse the resulting policies to show that the reciprocating agents are strongly influenced by their co-players' behavior.