Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning

TL;DR

This paper surveys recent methods in multi-agent deep reinforcement learning that address the challenge of non-stationarity caused by agents' evolving policies, highlighting various training and learning strategies.

Contribution

It provides a comprehensive overview of current techniques tackling non-stationarity in multi-agent deep reinforcement learning, including centralized training, opponent modeling, meta-learning, and communication.

Findings

Various methods mitigate non-stationarity in multi-agent settings

Centralized training improves coordination among agents

Open problems suggest future research directions

Abstract

Recent developments in deep reinforcement learning are concerned with creating decision-making agents which can perform well in various complex domains. A particular approach which has received increasing attention is multi-agent reinforcement learning, in which multiple agents learn concurrently to coordinate their actions. In such multi-agent environments, additional learning problems arise due to the continually changing decision-making policies of agents. This paper surveys recent works that address the non-stationarity problem in multi-agent deep reinforcement learning. The surveyed methods range from modifications in the training procedure, such as centralized training, to learning representations of the opponent's policy, meta-learning, communication, and decentralized learning. The survey concludes with a list of open problems and possible lines of future research.