Message-Dropout: An Efficient Training Method for Multi-Agent Deep Reinforcement Learning

TL;DR

This paper introduces message-dropout, a novel training method for multi-agent deep reinforcement learning that enhances robustness and performance by selectively dropping communication messages during training.

Contribution

The paper proposes message-dropout, a new technique that improves multi-agent reinforcement learning by handling communication errors and increasing training efficiency.

Findings

Message-dropout improves training speed.

It enhances steady-state performance.

It effectively manages communication errors.

Abstract

In this paper, we propose a new learning technique named message-dropout to improve the performance for multi-agent deep reinforcement learning under two application scenarios: 1) classical multi-agent reinforcement learning with direct message communication among agents and 2) centralized training with decentralized execution. In the first application scenario of multi-agent systems in which direct message communication among agents is allowed, the message-dropout technique drops out the received messages from other agents in a block-wise manner with a certain probability in the training phase and compensates for this effect by multiplying the weights of the dropped-out block units with a correction probability. The applied message-dropout technique effectively handles the increased input dimension in multi-agent reinforcement learning with communication and makes learning robust against communication errors in the execution phase. In the second application scenario of centralized training with decentralized execution, we particularly consider the application of the proposed message-dropout to Multi-Agent Deep Deterministic Policy Gradient (MADDPG), which uses a centralized critic to train a decentralized actor for each agent. We evaluate the proposed message-dropout technique for several games, and numerical results show that the proposed message-dropout technique with proper dropout rate improves the reinforcement learning performance significantly in terms of the training speed and the steady-state performance in the execution phase.