GAWM: Global-Aware World Model for Multi-Agent Reinforcement Learning

TL;DR

GAWM introduces a global-aware world model using Transformer architecture to improve sample efficiency and stability in multi-agent reinforcement learning, enabling better performance in complex environments.

Contribution

The paper proposes GAWM, a novel model-based MARL method that enhances global state representation with a Transformer, improving convergence and stability in complex multi-agent settings.

Findings

GAWM outperforms existing methods in SMAC benchmarks.

Enhanced global state representation improves training stability.

Method achieves superior convergence in complex environments.

Abstract

In recent years, Model-based Multi-Agent Reinforcement Learning (MARL) has demonstrated significant advantages over model-free methods in terms of sample efficiency by using independent environment dynamics world models for data sample augmentation. However, without considering the limited sample size, these methods still lag behind model-free methods in terms of final convergence performance and stability. This is primarily due to the world model's insufficient and unstable representation of global states in partially observable environments. This limitation hampers the ability to ensure global consistency in the data samples and results in a time-varying and unstable distribution mismatch between the pseudo data samples generated by the world model and the real samples. This issue becomes particularly pronounced in more complex multi-agent environments. To address this challenge, we propose a model-based MARL method called GAWM, which enhances the centralized world model's ability to achieve globally unified and accurate representation of state information while adhering to the CTDE paradigm. GAWM uniquely leverages an additional Transformer architecture to fuse local observation information from different agents, thereby improving its ability to extract and represent global state information. This enhancement not only improves sample efficiency but also enhances training stability, leading to superior convergence performance, particularly in complex and challenging multi-agent environments. This advancement enables model-based methods to be effectively applied to more complex multi-agent environments. Experimental results demonstrate that GAWM outperforms various model-free and model-based approaches, achieving exceptional performance in the challenging domains of SMAC.