Safe Multiagent Coordination via Entropic Exploration

TL;DR

This paper introduces E2C, a novel entropic exploration method for safe multiagent reinforcement learning that enhances exploration and safety by maximizing observation entropy, outperforming existing methods in complex tasks.

Contribution

It proposes a new entropic exploration approach for constrained multiagent RL, emphasizing joint team constraints and demonstrating improved safety and performance.

Findings

E2C matches or exceeds baseline performance.

E2C reduces unsafe behaviors by up to 50%.

E2C effectively balances exploration and safety.

Abstract

Many real-world multiagent learning problems involve safety concerns. In these setups, typical safe reinforcement learning algorithms constrain agents' behavior, limiting exploration -- a crucial component for discovering effective cooperative multiagent behaviors. Moreover, the multiagent literature typically models individual constraints for each agent and has yet to investigate the benefits of using joint team constraints. In this work, we analyze these team constraints from a theoretical and practical perspective and propose entropic exploration for constrained multiagent reinforcement learning (E2C) to address the exploration issue. E2C leverages observation entropy maximization to incentivize exploration and facilitate learning safe and effective cooperative behaviors. Experiments across increasingly complex domains show that E2C agents match or surpass common unconstrained and constrained baselines in task performance while reducing unsafe behaviors by up to $50\%$.