Emergent Reciprocity and Team Formation from Randomized Uncertain Social Preferences

TL;DR

This paper introduces RUSP, a novel environment augmentation for multi-agent reinforcement learning, enabling agents to develop social behaviors like reciprocity and team formation, improving cooperation in complex social dilemmas.

Contribution

The paper presents RUSP, a scalable environment augmentation that fosters emergent social behaviors in MARL without altering original game dynamics.

Findings

Emergent reciprocity and reputation behaviors observed

Higher social welfare equilibria achieved in social dilemmas

Applicable to various multi-agent environments

Abstract

Multi-agent reinforcement learning (MARL) has shown recent success in increasingly complex fixed-team zero-sum environments. However, the real world is not zero-sum nor does it have fixed teams; humans face numerous social dilemmas and must learn when to cooperate and when to compete. To successfully deploy agents into the human world, it may be important that they be able to understand and help in our conflicts. Unfortunately, selfish MARL agents typically fail when faced with social dilemmas. In this work, we show evidence of emergent direct reciprocity, indirect reciprocity and reputation, and team formation when training agents with randomized uncertain social preferences (RUSP), a novel environment augmentation that expands the distribution of environments agents play in. RUSP is generic and scalable; it can be applied to any multi-agent environment without changing the original underlying game dynamics or objectives. In particular, we show that with RUSP these behaviors can emerge and lead to higher social welfare equilibria in both classic abstract social dilemmas like Iterated Prisoner's Dilemma as well in more complex intertemporal environments.