Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a hierarchical message-passing approach for multi-agent reinforcement learning that combines hierarchical graph structures with a novel reward assignment method, improving coordination and scalability in complex environments.

Contribution

It presents a new hierarchical message-passing framework for multi-agent RL, integrating feudal HRL with a reward assignment strategy to enhance coordination and performance.

Findings

Outperforms state-of-the-art methods on benchmark tasks.

Effective hierarchical coordination among agents demonstrated.

Improves scalability and handling of partial observability.

Abstract

Decentralized Multi-Agent Reinforcement Learning (MARL) methods allow for learning scalable multi-agent policies, but suffer from partial observability and induced non-stationarity. These challenges can be addressed by introducing mechanisms that facilitate coordination and high-level planning. Specifically, coordination and temporal abstraction can be achieved through communication (e.g., message passing) and Hierarchical Reinforcement Learning (HRL) approaches to decision-making. However, optimization issues limit the applicability of hierarchical policies to multi-agent systems. As such, the combination of these approaches has not been fully explored. To fill this void, we propose a novel and effective methodology for learning multi-agent hierarchies of message-passing policies. We adopt the feudal HRL framework and rely on a hierarchical graph structure for planning and coordination among agents. Agents at lower levels in the hierarchy receive goals from the upper levels and exchange messages with neighboring agents at the same level. To learn hierarchical multi-agent policies, we design a novel reward-assignment method based on training the lower-level policies to maximize the advantage function associated with the upper levels. Results on relevant benchmarks show that our method performs favorably compared to the state of the art.