Experience Sharing Between Cooperative Reinforcement Learning Agents

TL;DR

This paper explores experience sharing strategies among cooperative reinforcement learning agents, proposing three novel methods that significantly accelerate learning compared to random sharing.

Contribution

It introduces three new experience sharing methods—Focused ES, Prioritized ES, and Focused Prioritized ES—that improve learning efficiency in cooperative multiagent reinforcement learning.

Findings

Focused ES doubles the learning speed

Proposed methods outperform random sharing baseline

51% reduction in episodes needed to complete task

Abstract

The idea of experience sharing between cooperative agents naturally emerges from our understanding of how humans learn. Our evolution as a species is tightly linked to the ability to exchange learned knowledge with one another. It follows that experience sharing (ES) between autonomous and independent agents could become the key to accelerate learning in cooperative multiagent settings. We investigate if randomly selecting experiences to share can increase the performance of deep reinforcement learning agents, and propose three new methods for selecting experiences to accelerate the learning process. Firstly, we introduce Focused ES, which prioritizes unexplored regions of the state space. Secondly, we present Prioritized ES, in which temporal-difference error is used as a measure of priority. Finally, we devise Focused Prioritized ES, which combines both previous approaches. The methods are empirically validated in a control problem. While sharing randomly selected experiences between two Deep Q-Network agents shows no improvement over a single agent baseline, we show that the proposed ES methods can successfully outperform the baseline. In particular, the Focused ES accelerates learning by a factor of 2, reducing by 51% the number of episodes required to complete the task.