Beyond Local Views: Global State Inference with Diffusion Models for Cooperative Multi-Agent Reinforcement Learning

TL;DR

This paper introduces SIDIFF, a diffusion model-based method that reconstructs global states from local observations in multi-agent systems, enhancing decision-making and performance in decentralized reinforcement learning.

Contribution

The paper presents SIDIFF, a novel diffusion model approach for global state inference from local observations, improving multi-agent reinforcement learning performance.

Findings

SIDIFF outperforms existing algorithms in multi-agent environments.

SIDIFF effectively reconstructs global states from local observations.

The method integrates seamlessly with current reinforcement learning algorithms.

Abstract

In partially observable multi-agent systems, agents typically only have access to local observations. This severely hinders their ability to make precise decisions, particularly during decentralized execution. To alleviate this problem and inspired by image outpainting, we propose State Inference with Diffusion Models (SIDIFF), which uses diffusion models to reconstruct the original global state based solely on local observations. SIDIFF consists of a state generator and a state extractor, which allow agents to choose suitable actions by considering both the reconstructed global state and local observations. In addition, SIDIFF can be effortlessly incorporated into current multi-agent reinforcement learning algorithms to improve their performance. Finally, we evaluated SIDIFF on different experimental platforms, including Multi-Agent Battle City (MABC), a novel and flexible multi-agent reinforcement learning environment we developed. SIDIFF achieved desirable results and outperformed other popular algorithms.