Learning to Balance Altruism and Self-interest Based on Empathy in Mixed-Motive Games

TL;DR

This paper introduces LASE Learning, a multi-agent reinforcement learning algorithm that balances altruism and self-interest by dynamically adapting gift-giving based on inferred social relationships, enhancing cooperation and fairness in mixed-motive games.

Contribution

The paper proposes a novel distributed RL algorithm that modulates altruistic behavior through social relationship inference, improving cooperation and protection against exploitation.

Findings

LASE promotes group collaboration without sacrificing fairness.

It adapts policies effectively to different co-player types.

Demonstrates success in spatially and temporally extended mixed-motive games.

Abstract

Real-world multi-agent scenarios often involve mixed motives, demanding altruistic agents capable of self-protection against potential exploitation. However, existing approaches often struggle to achieve both objectives. In this paper, based on that empathic responses are modulated by inferred social relationships between agents, we propose LASE Learning to balance Altruism and Self-interest based on Empathy), a distributed multi-agent reinforcement learning algorithm that fosters altruistic cooperation through gifting while avoiding exploitation by other agents in mixed-motive games. LASE allocates a portion of its rewards to co-players as gifts, with this allocation adapting dynamically based on the social relationship -- a metric evaluating the friendliness of co-players estimated by counterfactual reasoning. In particular, social relationship measures each co-player by comparing the estimated $Q$-function of current joint action to a counterfactual baseline which marginalizes the co-player's action, with its action distribution inferred by a perspective-taking module. Comprehensive experiments are performed in spatially and temporally extended mixed-motive games, demonstrating LASE's ability to promote group collaboration without compromising fairness and its capacity to adapt policies to various types of interactive co-players.