A Cooperation Graph Approach for Multiagent Sparse Reward Reinforcement Learning

TL;DR

This paper introduces a novel Cooperation Graph framework and a corresponding reinforcement learning algorithm to improve multiagent cooperation efficiency in sparse reward environments, achieving state-of-the-art results.

Contribution

The paper proposes a new Cooperation Graph structure and a CG-MARL algorithm that effectively addresses sparse reward challenges in multiagent reinforcement learning.

Findings

CG-MARL outperforms existing methods in benchmark tasks

The Cooperation Graph enables implicit cooperation among agents

Hierarchical graph control improves learning efficiency

Abstract

Multiagent reinforcement learning (MARL) can solve complex cooperative tasks. However, the efficiency of existing MARL methods relies heavily on well-defined reward functions. Multiagent tasks with sparse reward feedback are especially challenging not only because of the credit distribution problem, but also due to the low probability of obtaining positive reward feedback. In this paper, we design a graph network called Cooperation Graph (CG). The Cooperation Graph is the combination of two simple bipartite graphs, namely, the Agent Clustering subgraph (ACG) and the Cluster Designating subgraph (CDG). Next, based on this novel graph structure, we propose a Cooperation Graph Multiagent Reinforcement Learning (CG-MARL) algorithm, which can efficiently deal with the sparse reward problem in multiagent tasks. In CG-MARL, agents are directly controlled by the Cooperation Graph. And a policy neural network is trained to manipulate this Cooperation Graph, guiding agents to achieve cooperation in an implicit way. This hierarchical feature of CG-MARL provides space for customized cluster-actions, an extensible interface for introducing fundamental cooperation knowledge. In experiments, CG-MARL shows state-of-the-art performance in sparse reward multiagent benchmarks, including the anti-invasion interception task and the multi-cargo delivery task.