Reliably Re-Acting to Partner's Actions with the Social Intrinsic Motivation of Transfer Empowerment

TL;DR

This paper introduces transfer empowerment, a method to enhance multi-agent reinforcement learning by promoting reactive strategies, thereby improving cooperation and coordination among agents in simulated scenarios.

Contribution

It proposes transfer empowerment as a novel principle to bias MARL towards reactiveness, addressing overfitting to partner policies and improving coordination.

Findings

Transfer empowerment improves MARL performance in simulated cooperation tasks.

Agents trained with transfer empowerment are more reactive to partner actions.

The method supports better multi-agent coordination by ensuring influence-based reactivity.

Abstract

We consider multi-agent reinforcement learning (MARL) for cooperative communication and coordination tasks. MARL agents can be brittle because they can overfit their training partners' policies. This overfitting can produce agents that adopt policies that act under the expectation that other agents will act in a certain way rather than react to their actions. Our objective is to bias the learning process towards finding reactive strategies towards other agents' behaviors. Our method, transfer empowerment, measures the potential influence between agents' actions. Results from three simulated cooperation scenarios support our hypothesis that transfer empowerment improves MARL performance. We discuss how transfer empowerment could be a useful principle to guide multi-agent coordination by ensuring reactiveness to one's partner.