Coach-assisted Multi-Agent Reinforcement Learning Framework for Unexpected Crashed Agents

TL;DR

This paper introduces a novel coach-assisted multi-agent reinforcement learning framework that enhances robustness against unexpected agent crashes, addressing a critical gap between simulation and real-world multi-agent systems.

Contribution

It is the first to formally address unexpected crashes in multi-agent reinforcement learning and proposes a coach-assisted framework with strategies to improve system robustness.

Findings

The framework improves performance under crash conditions.

Adaptive strategies outperform fixed crash rate strategies.

Re-sampling enhances crash robustness.

Abstract

Multi-agent reinforcement learning is difficult to be applied in practice, which is partially due to the gap between the simulated and real-world scenarios. One reason for the gap is that the simulated systems always assume that the agents can work normally all the time, while in practice, one or more agents may unexpectedly "crash" during the coordination process due to inevitable hardware or software failures. Such crashes will destroy the cooperation among agents, leading to performance degradation. In this work, we present a formal formulation of a cooperative multi-agent reinforcement learning system with unexpected crashes. To enhance the robustness of the system to crashes, we propose a coach-assisted multi-agent reinforcement learning framework, which introduces a virtual coach agent to adjust the crash rate during training. We design three coaching strategies and the re-sampling strategy for our coach agent. To the best of our knowledge, this work is the first to study the unexpected crashes in the multi-agent system. Extensive experiments on grid-world and StarCraft II micromanagement tasks demonstrate the efficacy of adaptive strategy compared with the fixed crash rate strategy and curriculum learning strategy. The ablation study further illustrates the effectiveness of our re-sampling strategy.