On Memory Mechanism in Multi-Agent Reinforcement Learning

TL;DR

This paper investigates the role of memory mechanisms in multi-agent reinforcement learning, demonstrating their usefulness in modeling other agents and communication constraints, while cautioning against unintended memory solutions.

Contribution

It provides a comprehensive analysis of when and how memory mechanisms benefit multi-agent RL, including new experimental insights and considerations for effective implementation.

Findings

Memory helps in modeling other agents and communication constraints.

Agents can develop effective memory through alternative means, complicating analysis.

Memory mechanisms are particularly useful in environments with partial observability.

Abstract

Multi-agent reinforcement learning (MARL) extends (single-agent) reinforcement learning (RL) by introducing additional agents and (potentially) partial observability of the environment. Consequently, algorithms for solving MARL problems incorporate various extensions beyond traditional RL methods, such as a learned communication protocol between cooperative agents that enables exchange of private information or adaptive modeling of opponents in competitive settings. One popular algorithmic construct is a memory mechanism such that an agent's decisions can depend not only upon the current state but also upon the history of observed states and actions. In this paper, we study how a memory mechanism can be useful in environments with different properties, such as observability, internality and presence of a communication channel. Using both prior work and new experiments, we show that a memory mechanism is helpful when learning agents need to model other agents and/or when communication is constrained in some way; however we must to be cautious of agents achieving effective memoryfulness through other means.