Learning Cooperation and Online Planning Through Simulation and Graph Convolutional Network

TL;DR

This paper introduces SiCLOP, a novel online planning algorithm for multi-agent cooperative environments that combines MCTS, GCN, and CG to learn cooperation, adapt to dynamic environments, and improve scalability, outperforming existing methods.

Contribution

The paper presents SiCLOP, a new simulation-based online planning algorithm that integrates GCN and CG with MCTS, supporting transfer learning and real-time solutions in dynamic multi-agent settings.

Findings

SiCLOP outperforms state-of-the-art online planning algorithms.

Supports transfer learning across different environments.

Provides theoretical convergence analysis for multi-agent scenarios.

Abstract

Multi-agent Markov Decision Process (MMDP) has been an effective way of modelling sequential decision making algorithms for multi-agent cooperative environments. A number of algorithms based on centralized and decentralized planning have been developed in this domain. However, dynamically changing environment, coupled with exponential size of the state and joint action space, make it difficult for these algorithms to provide both efficiency and scalability. Recently, Centralized planning algorithm FV-MCTS-MP and decentralized planning algorithm \textit{Alternate maximization with Behavioural Cloning} (ABC) have achieved notable performance in solving MMDPs. However, they are not capable of adapting to dynamically changing environments and accounting for the lack of communication among agents, respectively. Against this background, we introduce a simulation based online planning algorithm, that we call SiCLOP, for multi-agent cooperative environments. Specifically, SiCLOP tailors Monte Carlo Tree Search (MCTS) and uses Coordination Graph (CG) and Graph Neural Network (GCN) to learn cooperation and provides real time solution of a MMDP problem. It also improves scalability through an effective pruning of action space. Additionally, unlike FV-MCTS-MP and ABC, SiCLOP supports transfer learning, which enables learned agents to operate in different environments. We also provide theoretical discussion about the convergence property of our algorithm within the context of multi-agent settings. Finally, our extensive empirical results show that SiCLOP significantly outperforms the state-of-the-art online planning algorithms.