A Privacy-preserving Distributed Training Framework for Cooperative Multi-agent Deep Reinforcement Learning

TL;DR

This paper introduces a privacy-preserving distributed training framework for multi-agent deep reinforcement learning, enabling agents to share knowledge without exposing raw data, leading to faster convergence and improved performance.

Contribution

It proposes a novel DNN architecture with global and local components and a distributed training framework that maintains privacy while facilitating collaboration among agents.

Findings

Enhanced convergence rate in multi-agent training

Improved performance in similar environments

Effective knowledge sharing without raw data exchange

Abstract

Deep Reinforcement Learning (DRL) sometimes needs a large amount of data to converge in the training procedure and in some cases, each action of the agent may produce regret. This barrier naturally motivates different data sets or environment owners to cooperate to share their knowledge and train their agents more efficiently. However, it raises privacy concerns if we directly merge the raw data from different owners. To solve this problem, we proposed a new Deep Neural Network (DNN) architecture with both global NN and local NN, and a distributed training framework. We allow the global weights to be updated by all the collaborator agents while the local weights are only updated by the agent they belong to. In this way, we hope the global weighs can share the common knowledge among these collaborators while the local NN can keep the specialized properties and ensure the agent to be compatible with its specific environment. Experiments show that the framework can efficiently help agents in the same or similar environments to collaborate in their training process and gain a higher convergence rate and better performance.