Paused Agent Replay Refresh

TL;DR

Paused Agent Replay Refresh (PARR) is a novel method that replaces traditional target networks in reinforcement learning, enabling more complex algorithms without approximation, leading to improved performance on challenging Atari games.

Contribution

PARR introduces a drop-in replacement for target networks that supports complex learning algorithms without approximation, enhancing reinforcement learning effectiveness.

Findings

Achieved 2500 points in Montezuma's Revenge after 30.9 million frames

Supports more complex algorithms without increased approximation

Offers a new perspective on sleep in carbon-based learning

Abstract

Reinforcement learning algorithms have become more complex since the invention of target networks. Unfortunately, target networks have not kept up with this increased complexity, instead requiring approximate solutions to be computationally feasible. These approximations increase noise in the Q-value targets and in the replay sampling distribution. Paused Agent Replay Refresh (PARR) is a drop-in replacement for target networks that supports more complex learning algorithms without this need for approximation. Using a basic Q-network architecture, and refreshing the novelty values, target values, and replay sampling distribution, PARR gets 2500 points in Montezuma's Revenge after only 30.9 million Atari frames. Finally, interpreting PARR in the context of carbon-based learning offers a new reason for sleep.