Efficient Episodic Memory Utilization of Cooperative Multi-Agent Reinforcement Learning

TL;DR

This paper introduces EMU, a novel method for cooperative multi-agent reinforcement learning that leverages semantic episodic memory and a new reward structure to accelerate learning and avoid local optima, demonstrated in complex environments.

Contribution

The paper proposes EMU, combining semantic episodic memory with a new reward incentive, to improve learning efficiency and policy quality in multi-agent reinforcement learning.

Findings

EMU accelerates learning in complex tasks.

EMU outperforms conventional methods in StarCraft II.

The episodic incentive improves policy convergence.

Abstract

In cooperative multi-agent reinforcement learning (MARL), agents aim to achieve a common goal, such as defeating enemies or scoring a goal. Existing MARL algorithms are effective but still require significant learning time and often get trapped in local optima by complex tasks, subsequently failing to discover a goal-reaching policy. To address this, we introduce Efficient episodic Memory Utilization (EMU) for MARL, with two primary objectives: (a) accelerating reinforcement learning by leveraging semantically coherent memory from an episodic buffer and (b) selectively promoting desirable transitions to prevent local convergence. To achieve (a), EMU incorporates a trainable encoder/decoder structure alongside MARL, creating coherent memory embeddings that facilitate exploratory memory recall. To achieve (b), EMU introduces a novel reward structure called episodic incentive based on the desirability of states. This reward improves the TD target in Q-learning and acts as an additional incentive for desirable transitions. We provide theoretical support for the proposed incentive and demonstrate the effectiveness of EMU compared to conventional episodic control. The proposed method is evaluated in StarCraft II and Google Research Football, and empirical results indicate further performance improvement over state-of-the-art methods.