Learning Existing Social Conventions via Observationally Augmented Self-Play

TL;DR

This paper proposes a method to help artificial agents learn social conventions by combining multi-agent reinforcement learning with imitation learning, enabling better coordination with existing groups in various environments.

Contribution

It introduces an augmentation technique for MARL using imitation learning to improve agents' alignment with real-world social conventions.

Findings

Augmentation with imitation learning increases convention alignment.

Method works in traffic, communication, and team coordination environments.

Enhances MARL effectiveness even when standard methods fail to find true conventions.

Abstract

In order for artificial agents to coordinate effectively with people, they must act consistently with existing conventions (e.g. how to navigate in traffic, which language to speak, or how to coordinate with teammates). A group's conventions can be viewed as a choice of equilibrium in a coordination game. We consider the problem of an agent learning a policy for a coordination game in a simulated environment and then using this policy when it enters an existing group. When there are multiple possible conventions we show that learning a policy via multi-agent reinforcement learning (MARL) is likely to find policies which achieve high payoffs at training time but fail to coordinate with the real group into which the agent enters. We assume access to a small number of samples of behavior from the true convention and show that we can augment the MARL objective to help it find policies consistent with the real group's convention. In three environments from the literature - traffic, communication, and team coordination - we observe that augmenting MARL with a small amount of imitation learning greatly increases the probability that the strategy found by MARL fits well with the existing social convention. We show that this works even in an environment where standard training methods very rarely find the true convention of the agent's partners.