ACCNet: Actor-Coordinator-Critic Net for "Learning-to-Communicate" with Deep Multi-agent Reinforcement Learning

TL;DR

This paper introduces ACCNet, a deep reinforcement learning framework that enables multi-agent systems to learn effective communication protocols from scratch in partially observable environments, outperforming existing baselines.

Contribution

The paper presents ACCNet, a novel actor-coordinator-critic architecture that integrates deep learning with reinforcement learning to learn communication protocols without predefined schemes.

Findings

ACCNet outperforms baseline methods in various environments.

It effectively learns communication protocols from scratch.

The learned protocols are interpretable and adaptable.

Abstract

Communication is a critical factor for the big multi-agent world to stay organized and productive. Typically, most previous multi-agent "learning-to-communicate" studies try to predefine the communication protocols or use technologies such as tabular reinforcement learning and evolutionary algorithm, which can not generalize to changing environment or large collection of agents. In this paper, we propose an Actor-Coordinator-Critic Net (ACCNet) framework for solving "learning-to-communicate" problem. The ACCNet naturally combines the powerful actor-critic reinforcement learning technology with deep learning technology. It can efficiently learn the communication protocols even from scratch under partially observable environment. We demonstrate that the ACCNet can achieve better results than several baselines under both continuous and discrete action space environments. We also analyse the learned protocols and discuss some design considerations.