MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimization

TL;DR

This paper introduces MAC-PO, a novel prioritized experience replay method for multi-agent reinforcement learning that optimizes sampling weights through regret minimization, leading to improved training efficiency and performance.

Contribution

We propose MAC-PO, which formulates and solves the optimal prioritized experience replay problem for multi-agent RL using a regret minimization framework and closed-form solutions.

Findings

MAC-PO outperforms state-of-the-art baselines in Predator-Prey.

It effectively replays important transitions, improving learning stability.

Experimental results show enhanced performance in StarCraft Multi-Agent Challenge.

Abstract

Experience replay is crucial for off-policy reinforcement learning (RL) methods. By remembering and reusing the experiences from past different policies, experience replay significantly improves the training efficiency and stability of RL algorithms. Many decision-making problems in practice naturally involve multiple agents and require multi-agent reinforcement learning (MARL) under centralized training decentralized execution paradigm. Nevertheless, existing MARL algorithms often adopt standard experience replay where the transitions are uniformly sampled regardless of their importance. Finding prioritized sampling weights that are optimized for MARL experience replay has yet to be explored. To this end, we propose MAC-PO, which formulates optimal prioritized experience replay for multi-agent problems as a regret minimization over the sampling weights of transitions. Such optimization is relaxed and solved using the Lagrangian multiplier approach to obtain the close-form optimal sampling weights. By minimizing the resulting policy regret, we can narrow the gap between the current policy and a nominal optimal policy, thus acquiring an improved prioritization scheme for multi-agent tasks. Our experimental results on Predator-Prey and StarCraft Multi-Agent Challenge environments demonstrate the effectiveness of our method, having a better ability to replay important transitions and outperforming other state-of-the-art baselines.