Count-Based Exploration with Neural Density Models

TL;DR

This paper enhances count-based exploration in reinforcement learning by using neural density models, specifically PixelCNN, to improve pseudo-count accuracy, and demonstrates significant performance gains on challenging Atari games.

Contribution

It introduces the use of PixelCNN for pseudo-counts in exploration, addressing model quality issues, and highlights the effectiveness of Monte Carlo updates in sparse reward environments.

Findings

PixelCNN pseudo-counts improve exploration performance

Mixed Monte Carlo updates facilitate exploration in sparse settings

State-of-the-art results on several Atari games

Abstract

Bellemare et al. (2016) introduced the notion of a pseudo-count, derived from a density model, to generalize count-based exploration to non-tabular reinforcement learning. This pseudo-count was used to generate an exploration bonus for a DQN agent and combined with a mixed Monte Carlo update was sufficient to achieve state of the art on the Atari 2600 game Montezuma's Revenge. We consider two questions left open by their work: First, how important is the quality of the density model for exploration? Second, what role does the Monte Carlo update play in exploration? We answer the first question by demonstrating the use of PixelCNN, an advanced neural density model for images, to supply a pseudo-count. In particular, we examine the intrinsic difficulties in adapting Bellemare et al.'s approach when assumptions about the model are violated. The result is a more practical and general algorithm requiring no special apparatus. We combine PixelCNN pseudo-counts with different agent architectures to dramatically improve the state of the art on several hard Atari games. One surprising finding is that the mixed Monte Carlo update is a powerful facilitator of exploration in the sparsest of settings, including Montezuma's Revenge.