Multi-agent Reinforcement Learning for Networked System Control

TL;DR

This paper advances multi-agent reinforcement learning for networked control systems by introducing a spatial discount factor and a novel communication protocol, NeurComm, improving training stability and performance in traffic and cruise control scenarios.

Contribution

It proposes a new NMARL framework with a spatial discount factor and NeurComm protocol, enhancing learning stability and communication efficiency.

Findings

Spatial discount factor improves learning curves.

NeurComm outperforms existing protocols.

Enhanced control performance in traffic and cruise scenarios.

Abstract

This paper considers multi-agent reinforcement learning (MARL) in networked system control. Specifically, each agent learns a decentralized control policy based on local observations and messages from connected neighbors. We formulate such a networked MARL (NMARL) problem as a spatiotemporal Markov decision process and introduce a spatial discount factor to stabilize the training of each local agent. Further, we propose a new differentiable communication protocol, called NeurComm, to reduce information loss and non-stationarity in NMARL. Based on experiments in realistic NMARL scenarios of adaptive traffic signal control and cooperative adaptive cruise control, an appropriate spatial discount factor effectively enhances the learning curves of non-communicative MARL algorithms, while NeurComm outperforms existing communication protocols in both learning efficiency and control performance.