ROMAX: Certifiably Robust Deep Multiagent Reinforcement Learning via Convex Relaxation

TL;DR

ROMAX introduces a convex relaxation-based minimax approach to enhance robustness in multiagent reinforcement learning, providing certified bounds and improved performance against cyber-physical attacks in multi-robot systems.

Contribution

The paper proposes a novel convex relaxation technique for certifiably robust multiagent reinforcement learning, addressing non-stationarity and adversarial robustness.

Findings

Outperforms previous state-of-the-art methods on mixed cooperative-competitive tasks.

Provides certified robustness bounds for multiagent policies.

Demonstrates effectiveness against cyber-physical attacks in multirobot systems.

Abstract

In a multirobot system, a number of cyber-physical attacks (e.g., communication hijack, observation perturbations) can challenge the robustness of agents. This robustness issue worsens in multiagent reinforcement learning because there exists the non-stationarity of the environment caused by simultaneously learning agents whose changing policies affect the transition and reward functions. In this paper, we propose a minimax MARL approach to infer the worst-case policy update of other agents. As the minimax formulation is computationally intractable to solve, we apply the convex relaxation of neural networks to solve the inner minimization problem. Such convex relaxation enables robustness in interacting with peer agents that may have significantly different behaviors and also achieves a certified bound of the original optimization problem. We evaluate our approach on multiple mixed cooperative-competitive tasks and show that our method outperforms the previous state of the art approaches on this topic.