Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research

TL;DR

This paper reevaluates the Rainbow deep reinforcement learning algorithm on traditional small-scale environments to promote more inclusive research and gain new insights, highlighting the value of accessible benchmarks.

Contribution

It empirically revisits Rainbow on small environments, demonstrating their continued scientific value and advocating for more inclusive RL research practices.

Findings

Small environments can still provide valuable insights into RL algorithms.

Revisiting Rainbow yields new understanding of its components.

Accessible benchmarks can democratize RL research.

Abstract

Since the introduction of DQN, a vast majority of reinforcement learning research has focused on reinforcement learning with deep neural networks as function approximators. New methods are typically evaluated on a set of environments that have now become standard, such as Atari 2600 games. While these benchmarks help standardize evaluation, their computational cost has the unfortunate side effect of widening the gap between those with ample access to computational resources, and those without. In this work we argue that, despite the community's emphasis on large-scale environments, the traditional small-scale environments can still yield valuable scientific insights and can help reduce the barriers to entry for underprivileged communities. To substantiate our claims, we empirically revisit the paper which introduced the Rainbow algorithm [Hessel et al., 2018] and present some new insights into the algorithms used by Rainbow.