Prosocial learning agents solve generalized Stag Hunts better than selfish ones

TL;DR

This paper demonstrates that making a single reinforcement learning agent prosocial by caring about its partners' rewards can significantly improve group outcomes in Stag Hunt environments, including complex scenarios with raw inputs.

Contribution

The study introduces a simple modification to reward functions, making one agent prosocial, which enhances convergence to optimal outcomes in multi-agent Stag Hunt games.

Findings

Prosocial agents increase likelihood of cooperative convergence.

Single-agent prosociality improves long-term group payoffs.

Results extend to complex environments with raw pixel inputs.

Abstract

Deep reinforcement learning has become an important paradigm for constructing agents that can enter complex multi-agent situations and improve their policies through experience. One commonly used technique is reactive training - applying standard RL methods while treating other agents as a part of the learner's environment. It is known that in general-sum games reactive training can lead groups of agents to converge to inefficient outcomes. We focus on one such class of environments: Stag Hunt games. Here agents either choose a risky cooperative policy (which leads to high payoffs if both choose it but low payoffs to an agent who attempts it alone) or a safe one (which leads to a safe payoff no matter what). We ask how we can change the learning rule of a single agent to improve its outcomes in Stag Hunts that include other reactive learners. We extend existing work on reward-shaping in multi-agent reinforcement learning and show that that making a single agent prosocial, that is, making them care about the rewards of their partners can increase the probability that groups converge to good outcomes. Thus, even if we control a single agent in a group making that agent prosocial can increase our agent's long-run payoff. We show experimentally that this result carries over to a variety of more complex environments with Stag Hunt-like dynamics including ones where agents must learn from raw input pixels.