Negative Update Intervals in Deep Multi-Agent Reinforcement Learning

TL;DR

This paper introduces NUI-DDQN, a new deep multi-agent reinforcement learning algorithm that effectively overcomes multiple learning pathologies in complex environments by discarding episodes with outlier rewards.

Contribution

The paper presents NUI-DDQN, a novel method that improves multi-agent learning by filtering out episodes with misleading reward signals, outperforming existing approaches in complex scenarios.

Findings

NUI-DDQN reliably learns near-optimal policies in complex environments.

Existing lenient and hysteretic methods often fail in complex multi-agent settings.

NUI-DDQN overcomes multiple learning pathologies simultaneously.

Abstract

In Multi-Agent Reinforcement Learning (MA-RL), independent cooperative learners must overcome a number of pathologies to learn optimal joint policies. Addressing one pathology often leaves approaches vulnerable towards others. For instance, hysteretic Q-learning addresses miscoordination while leaving agents vulnerable towards misleading stochastic rewards. Other methods, such as leniency, have proven more robust when dealing with multiple pathologies simultaneously. However, leniency has predominately been studied within the context of strategic form games (bimatrix games) and fully observable Markov games consisting of a small number of probabilistic state transitions. This raises the question of whether these findings scale to more complex domains. For this purpose we implement a temporally extend version of the Climb Game, within which agents must overcome multiple pathologies simultaneously, including relative overgeneralisation, stochasticity, the alter-exploration and moving target problems, while learning from a large observation space. We find that existing lenient and hysteretic approaches fail to consistently learn near optimal joint-policies in this environment. To address these pathologies we introduce Negative Update Intervals-DDQN (NUI-DDQN), a Deep MA-RL algorithm which discards episodes yielding cumulative rewards outside the range of expanding intervals. NUI-DDQN consistently gravitates towards optimal joint-policies in our environment, overcoming the outlined pathologies.