Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning

TL;DR

This paper introduces a novel count-based exploration method in reinforcement learning that uses coin flip sampling to estimate state visitation counts, outperforming previous density-based approaches especially in complex environments like Atari games.

Contribution

The authors present a simple supervised learning approach using coin flips to estimate pseudocounts, improving exploration efficiency in high-dimensional state spaces.

Findings

Outperforms previous count estimation methods in accuracy.

Achieves better exploration results on challenging Atari games.

Significantly improves reinforcement learning performance in complex tasks.

Abstract

We propose a new method for count-based exploration in high-dimensional state spaces. Unlike previous work which relies on density models, we show that counts can be derived by averaging samples from the Rademacher distribution (or coin flips). This insight is used to set up a simple supervised learning objective which, when optimized, yields a state's visitation count. We show that our method is significantly more effective at deducing ground-truth visitation counts than previous work; when used as an exploration bonus for a model-free reinforcement learning algorithm, it outperforms existing approaches on most of 9 challenging exploration tasks, including the Atari game Montezuma's Revenge.