Deep Meta Coordination Graphs for Multi-agent Reinforcement Learning

TL;DR

This paper introduces deep meta coordination graphs (DMCG) that dynamically learn expressive agent interaction representations to enhance cooperation and efficiency in multi-agent reinforcement learning tasks.

Contribution

The paper proposes a novel DMCG framework that learns evolving coordination graphs for better multi-agent cooperation, integrating graph convolutional networks with joint value function optimization.

Findings

DMCG achieves state-of-the-art performance on cooperative tasks.

It improves sample efficiency compared to prior methods.

Ablation studies confirm the effectiveness of key components.

Abstract

This paper presents deep meta coordination graphs (DMCG) for learning cooperative policies in multi-agent reinforcement learning (MARL). Coordination graph formulations encode local interactions and accordingly factorize the joint value function of all agents to improve efficiency in MARL. Through DMCG, we dynamically compose what we refer to as \textit{meta coordination graphs}, to learn a more expressive representation of agent interactions and use them to integrate agent information through graph convolutional networks. The goal is to enable an evolving coordination graph to guide effective coordination in cooperative MARL tasks. The graphs are jointly optimized with agents' value functions to learn to implicitly reason about joint actions, facilitating the end-to-end learning of interaction representations and coordinated policies. We demonstrate that DMCG consistently achieves state-of-the-art coordination performance and sample efficiency on challenging cooperative tasks, outperforming several prior graph-based and non-graph-based MARL baselines. Through several ablations, we also isolate the impact of individual components in DMCG, showing that the observed improvements are due to the meaningful design choices in this approach. We also include an analysis of its computational complexity to discuss its practicality in real-world applications. All codes can be found here: {\color{blue}{https://github.com/Nikunj-Gupta/dmcg-marl}.