Interaction Pattern Disentangling for Multi-Agent Reinforcement Learning

TL;DR

This paper introduces OPT, a novel method for disentangling interaction patterns in multi-agent reinforcement learning, improving interpretability and generalization by filtering noisy interactions and capturing underlying interaction prototypes.

Contribution

OPT is the first approach to explicitly disentangle interaction patterns into prototypes, enhancing interpretability and robustness in multi-agent reinforcement learning.

Findings

OPT outperforms state-of-the-art methods on various benchmarks.

The sparse disagreement mechanism promotes diversity among prototypes.

Maximizing mutual information stabilizes training under partial observability.

Abstract

Deep cooperative multi-agent reinforcement learning has demonstrated its remarkable success over a wide spectrum of complex control tasks. However, recent advances in multi-agent learning mainly focus on value decomposition while leaving entity interactions still intertwined, which easily leads to over-fitting on noisy interactions between entities. In this work, we introduce a novel interactiOn Pattern disenTangling (OPT) method, to disentangle the entity interactions into interaction prototypes, each of which represents an underlying interaction pattern within a subgroup of the entities. OPT facilitates filtering the noisy interactions between irrelevant entities and thus significantly improves generalizability as well as interpretability. Specifically, OPT introduces a sparse disagreement mechanism to encourage sparsity and diversity among discovered interaction prototypes. Then the model selectively restructures these prototypes into a compact interaction pattern by an aggregator with learnable weights. To alleviate the training instability issue caused by partial observability, we propose to maximize the mutual information between the aggregation weights and the history behaviors of each agent. Experiments on single-task, multi-task and zero-shot benchmarks demonstrate that the proposed method yields results superior to the state-of-the-art counterparts. Our code is available at https://github.com/liushunyu/OPT.