Actor Prioritized Experience Replay

TL;DR

This paper identifies limitations of Prioritized Experience Replay in actor-critic methods for continuous control, introduces a new sampling framework, and demonstrates significant performance improvements and state-of-the-art results.

Contribution

It provides a theoretical analysis of PER's shortcomings in actor-critic algorithms and proposes a novel experience replay method tailored for continuous control tasks.

Findings

Theoretical proof that large TD errors hinder actor network training.

The new sampling framework improves stability and training efficiency.

Achieves state-of-the-art results on continuous control benchmarks.

Abstract

A widely-studied deep reinforcement learning (RL) technique known as Prioritized Experience Replay (PER) allows agents to learn from transitions sampled with non-uniform probability proportional to their temporal-difference (TD) error. Although it has been shown that PER is one of the most crucial components for the overall performance of deep RL methods in discrete action domains, many empirical studies indicate that it considerably underperforms actor-critic algorithms in continuous control. We theoretically show that actor networks cannot be effectively trained with transitions that have large TD errors. As a result, the approximate policy gradient computed under the Q-network diverges from the actual gradient computed under the optimal Q-function. Motivated by this, we introduce a novel experience replay sampling framework for actor-critic methods, which also regards issues with stability and recent findings behind the poor empirical performance of PER. The introduced algorithm suggests a new branch of improvements to PER and schedules effective and efficient training for both actor and critic networks. An extensive set of experiments verifies our theoretical claims and demonstrates that the introduced method significantly outperforms the competing approaches and obtains state-of-the-art results over the standard off-policy actor-critic algorithms.