Emergent Prosociality in Multi-Agent Games Through Gifting

TL;DR

This paper introduces a gifting mechanism in multi-agent reinforcement learning that promotes prosocial behavior and convergence to socially desirable equilibria without requiring agents to be explicitly prosocial.

Contribution

It proposes a peer-rewarding gifting mechanism and a theoretical framework to analyze its effect on equilibrium convergence in multi-agent systems.

Findings

Gifting increases convergence to prosocial equilibria in coordination games.

Numerical analysis shows improved stability with gifting.

Experiments confirm enhanced prosocial behavior through gifting.

Abstract

Coordination is often critical to forming prosocial behaviors -- behaviors that increase the overall sum of rewards received by all agents in a multi-agent game. However, state of the art reinforcement learning algorithms often suffer from converging to socially less desirable equilibria when multiple equilibria exist. Previous works address this challenge with explicit reward shaping, which requires the strong assumption that agents can be forced to be prosocial. We propose using a less restrictive peer-rewarding mechanism, gifting, that guides the agents toward more socially desirable equilibria while allowing agents to remain selfish and decentralized. Gifting allows each agent to give some of their reward to other agents. We employ a theoretical framework that captures the benefit of gifting in converging to the prosocial equilibrium by characterizing the equilibria's basins of attraction in a dynamical system. With gifting, we demonstrate increased convergence of high risk, general-sum coordination games to the prosocial equilibrium both via numerical analysis and experiments.