Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment

TL;DR

This paper empirically evaluates recent bonus-based exploration methods in the Arcade Learning Environment, finding limited improvements on challenging games like Montezuma's Revenge and potential negative impacts on easier games.

Contribution

It provides a systematic comparison of recent exploration bonuses using Rainbow across multiple Atari games, highlighting their limited effectiveness and potential drawbacks.

Findings

Recent bonuses do not significantly improve performance on hard exploration games.

Bonus methods may negatively affect performance on easier games.

Some bonuses perform worse than epsilon-greedy exploration.

Abstract

This paper provides an empirical evaluation of recently developed exploration algorithms within the Arcade Learning Environment (ALE). We study the use of different reward bonuses that incentives exploration in reinforcement learning. We do so by fixing the learning algorithm used and focusing only on the impact of the different exploration bonuses in the agent's performance. We use Rainbow, the state-of-the-art algorithm for value-based agents, and focus on some of the bonuses proposed in the last few years. We consider the impact these algorithms have on performance within the popular game Montezuma's Revenge which has gathered a lot of interest from the exploration community, across the the set of seven games identified by Bellemare et al. (2016) as challenging for exploration, and easier games where exploration is not an issue. We find that, in our setting, recently developed bonuses do not provide significantly improved performance on Montezuma's Revenge or hard exploration games. We also find that existing bonus-based methods may negatively impact performance on games in which exploration is not an issue and may even perform worse than $\epsilon$-greedy exploration.