Distributed Detection of Adversarial Attacks in Multi-Agent Reinforcement Learning with Continuous Action Space

TL;DR

This paper presents a decentralized, neural network-based detection method for identifying adversarial attacks in cooperative multi-agent reinforcement learning with continuous actions, demonstrating high accuracy and real-time detection capabilities.

Contribution

It introduces a novel decentralized detection approach using deep neural networks and statistical analysis for continuous action spaces in multi-agent RL, outperforming existing methods.

Findings

Achieves over 0.95 AUC-ROC scores against impactful attacks

Outperforms discrete counterparts in detection accuracy

Effective in real-time detection across various benchmarks

Abstract

We address the problem of detecting adversarial attacks against cooperative multi-agent reinforcement learning with continuous action space. We propose a decentralized detector that relies solely on the local observations of the agents and makes use of a statistical characterization of the normal behavior of observable agents. The proposed detector utilizes deep neural networks to approximate the normal behavior of agents as parametric multivariate Gaussian distributions. Based on the predicted density functions, we define a normality score and provide a characterization of its mean and variance. This characterization allows us to employ a two-sided CUSUM procedure for detecting deviations of the normality score from its mean, serving as a detector of anomalous behavior in real-time. We evaluate our scheme on various multi-agent PettingZoo benchmarks against different state-of-the-art attack methods, and our results demonstrate the effectiveness of our method in detecting impactful adversarial attacks. Particularly, it outperforms the discrete counterpart by achieving AUC-ROC scores of over 0.95 against the most impactful attacks in all evaluated environments.