Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Delayed Observation

TL;DR

This paper introduces Rainbow Delay Compensation (RDC), a novel multi-agent reinforcement learning framework designed to mitigate the adverse effects of observation delays in multi-agent systems, improving performance and robustness.

Contribution

The paper formulates a new DSID-POMDP model and proposes RDC, a training framework that effectively addresses stochastic observation delays in MARL environments.

Findings

RDC significantly reduces performance degradation caused by delays.

Baseline MARL methods perform poorly under delays, while RDC maintains near delay-free performance.

RDC demonstrates strong generalizability across different delay scenarios.

Abstract

In real-world multi-agent systems (MASs), observation delays are ubiquitous, preventing agents from making decisions based on the environment's true state. An individual agent's local observation typically comprises multiple components from other agents or dynamic entities within the environment. These discrete observation components with varying delay characteristics pose significant challenges for multi-agent reinforcement learning (MARL). In this paper, we first formulate the decentralized stochastic individual delay partially observable Markov decision process (DSID-POMDP) by extending the standard Dec-POMDP. We then propose the Rainbow Delay Compensation (RDC), a MARL training framework for addressing stochastic individual delays, along with recommended implementations for its constituent modules. We implement the DSID-POMDP's observation generation pattern using standard MARL benchmarks, including MPE and SMAC. Experiments demonstrate that baseline MARL methods suffer severe performance degradation under fixed and unfixed delays. The RDC-enhanced approach mitigates this issue, remarkably achieving ideal delay-free performance in certain delay scenarios while maintaining generalizability. Our work provides a novel perspective on multi-agent delayed observation problems and offers an effective solution framework. The source code is available at https://github.com/linkjoker1006/RDC-pymarl.