Robust Multi-Agent Reinforcement Learning by Mutual Information Regularization

TL;DR

This paper introduces MIR3, a mutual information regularization approach for robust multi-agent reinforcement learning that improves safety and efficiency without explicit threat scenario enumeration, validated in StarCraft II and robot swarms.

Contribution

It proposes MIR3, a novel regularization technique that implicitly maximizes robustness in MARL through an information bottleneck, avoiding explicit threat modeling.

Findings

MIR3 outperforms baseline methods in robustness and efficiency.

MIR3 maintains cooperative performance in complex environments.

Real-world robot swarm deployment shows 14.29% improvement.

Abstract

In multi-agent reinforcement learning (MARL), ensuring robustness against unpredictable or worst-case actions by allies is crucial for real-world deployment. Existing robust MARL methods either approximate or enumerate all possible threat scenarios against worst-case adversaries, leading to computational intensity and reduced robustness. In contrast, human learning efficiently acquires robust behaviors in daily life without preparing for every possible threat. Inspired by this, we frame robust MARL as an inference problem, with worst-case robustness implicitly optimized under all threat scenarios via off-policy evaluation. Within this framework, we demonstrate that Mutual Information Regularization as Robust Regularization (MIR3) during routine training is guaranteed to maximize a lower bound on robustness, without the need for adversaries. Further insights show that MIR3 acts as an information bottleneck, preventing agents from over-reacting to others and aligning policies with robust action priors. In the presence of worst-case adversaries, our MIR3 significantly surpasses baseline methods in robustness and training efficiency while maintaining cooperative performance in StarCraft II and robot swarm control. When deploying the robot swarm control algorithm in the real world, our method also outperforms the best baseline by 14.29%.