RMIO: A Model-Based MARL Framework for Scenarios with Observation Loss in Some Agents

TL;DR

RMIO is a model-based multi-agent reinforcement learning framework that effectively reconstructs missing observations and enhances decision-making stability in scenarios with observation loss, outperforming existing methods.

Contribution

RMIO introduces a novel approach combining world model-based reconstruction, inter-agent information sharing, and limited communication to handle observation loss in MARL.

Findings

Outperforms state-of-the-art in SMAC and MaMuJoCo environments.

Achieves better asymptotic convergence and policy robustness.

Effectively handles observation loss scenarios.

Abstract

In recent years, model-based reinforcement learning (MBRL) has emerged as a solution to address sample complexity in multi-agent reinforcement learning (MARL) by modeling agent-environment dynamics to improve sample efficiency. However, most MBRL methods assume complete and continuous observations from each agent during the inference stage, which can be overly idealistic in practical applications. A novel model-based MARL approach called RMIO is introduced to address this limitation, specifically designed for scenarios where observation is lost in some agent. RMIO leverages the world model to reconstruct missing observations, and further reduces reconstruction errors through inter-agent information integration to ensure stable multi-agent decision-making. Secondly, unlike CTCE methods such as MAMBA, RMIO adopts the CTDE paradigm in standard environment, and enabling limited communication only when agents lack observation data, thereby reducing reliance on communication. Additionally, RMIO improves asymptotic performance through strategies such as reward smoothing, a dual-layer experience replay buffer, and an RNN-augmented policy model, surpassing previous work. Our experiments conducted in both the SMAC and MaMuJoCo environments demonstrate that RMIO outperforms current state-of-the-art approaches in terms of asymptotic convergence performance and policy robustness, both in standard mission settings and in scenarios involving observation loss.