DPMAC: Differentially Private Communication for Cooperative Multi-Agent Reinforcement Learning

TL;DR

This paper introduces DPMAC, a novel multi-agent reinforcement learning algorithm that ensures agents' communication privacy through differential privacy, maintaining cooperation while protecting sensitive information.

Contribution

The paper proposes a new differentially private communication method for MARL with a stochastic message sender and proves the existence of a Nash equilibrium under privacy constraints.

Findings

DPMAC outperforms baseline methods in privacy-preserving scenarios.

The stochastic message sender effectively balances privacy and communication quality.

Existence of Nash equilibrium ensures the theoretical viability of privacy-preserving MARL.

Abstract

Communication lays the foundation for cooperation in human society and in multi-agent reinforcement learning (MARL). Humans also desire to maintain their privacy when communicating with others, yet such privacy concern has not been considered in existing works in MARL. To this end, we propose the \textit{differentially private multi-agent communication} (DPMAC) algorithm, which protects the sensitive information of individual agents by equipping each agent with a local message sender with rigorous $(\epsilon, \delta)$-differential privacy (DP) guarantee. In contrast to directly perturbing the messages with predefined DP noise as commonly done in privacy-preserving scenarios, we adopt a stochastic message sender for each agent respectively and incorporate the DP requirement into the sender, which automatically adjusts the learned message distribution to alleviate the instability caused by DP noise. Further, we prove the existence of a Nash equilibrium in cooperative MARL with privacy-preserving communication, which suggests that this problem is game-theoretically learnable. Extensive experiments demonstrate a clear advantage of DPMAC over baseline methods in privacy-preserving scenarios.