Cooperative Policy Learning with Pre-trained Heterogeneous Observation Representations

TL;DR

This paper introduces a novel cooperative policy learning framework in multi-agent reinforcement learning that leverages pre-trained heterogeneous observation representations and graph attention mechanisms, improving performance in complex real-world scenarios.

Contribution

The paper proposes a new MARL framework using pre-trained heterogeneous representations and graph attention to better model complex interactions, easing training difficulties.

Findings

Significant performance improvements over existing MARL baselines.

Effective modeling of heterogeneous interactions with graph attention.

Enhanced scalability and robustness in real-world scenarios.

Abstract

Multi-agent reinforcement learning (MARL) has been increasingly explored to learn the cooperative policy towards maximizing a certain global reward. Many existing studies take advantage of graph neural networks (GNN) in MARL to propagate critical collaborative information over the interaction graph, built upon inter-connected agents. Nevertheless, the vanilla GNN approach yields substantial defects in dealing with complex real-world scenarios since the generic message passing mechanism is ineffective between heterogeneous vertices and, moreover, simple message aggregation functions are incapable of accurately modeling the combinational interactions from multiple neighbors. While adopting complex GNN models with more informative message passing and aggregation mechanisms can obviously benefit heterogeneous vertex representations and cooperative policy learning, it could, on the other hand, increase the training difficulty of MARL and demand more intense and direct reward signals compared to the original global reward. To address these challenges, we propose a new cooperative learning framework with pre-trained heterogeneous observation representations. Particularly, we employ an encoder-decoder based graph attention to learn the intricate interactions and heterogeneous representations that can be more easily leveraged by MARL. Moreover, we design a pre-training with local actor-critic algorithm to ease the difficulty in cooperative policy learning. Extensive experiments over real-world scenarios demonstrate that our new approach can significantly outperform existing MARL baselines as well as operational research solutions that are widely-used in industry.