Efficient Multi-agent Reinforcement Learning by Planning

TL;DR

This paper introduces MAZero, a model-based multi-agent reinforcement learning algorithm that combines planning and search techniques to improve sample efficiency and performance in large-scale decision-making tasks.

Contribution

The paper proposes MAZero, integrating a centralized model with Monte Carlo Tree Search and novel techniques for efficient multi-agent planning, advancing model-based MARL methods.

Findings

MAZero outperforms model-free methods in sample efficiency.

MAZero achieves comparable or better performance than existing model-based approaches.

The approach demonstrates effectiveness on the SMAC benchmark.

Abstract

Multi-agent reinforcement learning (MARL) algorithms have accomplished remarkable breakthroughs in solving large-scale decision-making tasks. Nonetheless, most existing MARL algorithms are model-free, limiting sample efficiency and hindering their applicability in more challenging scenarios. In contrast, model-based reinforcement learning (MBRL), particularly algorithms integrating planning, such as MuZero, has demonstrated superhuman performance with limited data in many tasks. Hence, we aim to boost the sample efficiency of MARL by adopting model-based approaches. However, incorporating planning and search methods into multi-agent systems poses significant challenges. The expansive action space of multi-agent systems often necessitates leveraging the nearly-independent property of agents to accelerate learning. To tackle this issue, we propose the MAZero algorithm, which combines a centralized model with Monte Carlo Tree Search (MCTS) for policy search. We design a novel network structure to facilitate distributed execution and parameter sharing. To enhance search efficiency in deterministic environments with sizable action spaces, we introduce two novel techniques: Optimistic Search Lambda (OS($\lambda$)) and Advantage-Weighted Policy Optimization (AWPO). Extensive experiments on the SMAC benchmark demonstrate that MAZero outperforms model-free approaches in terms of sample efficiency and provides comparable or better performance than existing model-based methods in terms of both sample and computational efficiency. Our code is available at https://github.com/liuqh16/MAZero.