Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning

TL;DR

This paper introduces Deep Implicit Coordination Graphs (DICG), a novel MARL architecture that infers dynamic coordination structures and uses graph neural networks to improve multi-agent cooperation in complex tasks.

Contribution

DICG is the first method to learn dynamic coordination graphs in MARL, enabling scalable reasoning about joint actions without domain-specific design.

Findings

DICG effectively solves relative overgeneralization in prey-predator tasks.

DICG outperforms baselines on StarCraft II Multi-agent Challenge.

DICG improves coordination in traffic junction environments.

Abstract

Multi-agent reinforcement learning (MARL) requires coordination to efficiently solve certain tasks. Fully centralized control is often infeasible in such domains due to the size of joint action spaces. Coordination graph based formalization allows reasoning about the joint action based on the structure of interactions. However, they often require domain expertise in their design. This paper introduces the deep implicit coordination graph (DICG) architecture for such scenarios. DICG consists of a module for inferring the dynamic coordination graph structure which is then used by a graph neural network based module to learn to implicitly reason about the joint actions or values. DICG allows learning the tradeoff between full centralization and decentralization via standard actor-critic methods to significantly improve coordination for domains with large number of agents. We apply DICG to both centralized-training-centralized-execution and centralized-training-decentralized-execution regimes. We demonstrate that DICG solves the relative overgeneralization pathology in predatory-prey tasks as well as outperforms various MARL baselines on the challenging StarCraft II Multi-agent Challenge (SMAC) and traffic junction environments.