Tackling Uncertainties in Multi-Agent Reinforcement Learning through Integration of Agent Termination Dynamics

TL;DR

This paper presents a novel MARL approach that combines distributional learning with safety-focused loss functions, improving convergence and safety in complex multi-agent tasks like StarCraft II.

Contribution

It introduces a Barrier Function based loss integrating safety metrics into MARL, addressing uncertainty and risk during policy learning.

Findings

Enhanced convergence in StarCraft II benchmarks

Outperforms state-of-the-art baselines in safety and task completion

Demonstrates the effectiveness of safety-aware learning in MARL

Abstract

Multi-Agent Reinforcement Learning (MARL) has gained significant traction for solving complex real-world tasks, but the inherent stochasticity and uncertainty in these environments pose substantial challenges to efficient and robust policy learning. While Distributional Reinforcement Learning has been successfully applied in single-agent settings to address risk and uncertainty, its application in MARL is substantially limited. In this work, we propose a novel approach that integrates distributional learning with a safety-focused loss function to improve convergence in cooperative MARL tasks. Specifically, we introduce a Barrier Function based loss that leverages safety metrics, identified from inherent faults in the system, into the policy learning process. This additional loss term helps mitigate risks and encourages safer exploration during the early stages of training. We evaluate our method in the StarCraft II micromanagement benchmark, where our approach demonstrates improved convergence and outperforms state-of-the-art baselines in terms of both safety and task completion. Our results suggest that incorporating safety considerations can significantly enhance learning performance in complex, multi-agent environments.