RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learning

TL;DR

RLeXplore is a comprehensive, modular framework that standardizes and facilitates research on intrinsic rewards in reinforcement learning, addressing implementation challenges and promoting progress in unsupervised learning environments.

Contribution

It introduces a unified, modular framework with implementations of eight intrinsic reward methods and provides standardized practices for their effective use.

Findings

Identified key implementation details affecting intrinsic reward performance.

Established best practices for intrinsically-motivated RL.

Provided a publicly available, extensible codebase.

Abstract

Extrinsic rewards can effectively guide reinforcement learning (RL) agents in specific tasks. However, extrinsic rewards frequently fall short in complex environments due to the significant human effort needed for their design and annotation. This limitation underscores the necessity for intrinsic rewards, which offer auxiliary and dense signals and can enable agents to learn in an unsupervised manner. Although various intrinsic reward formulations have been proposed, their implementation and optimization details are insufficiently explored and lack standardization, thereby hindering research progress. To address this gap, we introduce RLeXplore, a unified, highly modularized, and plug-and-play framework offering reliable implementations of eight state-of-the-art intrinsic reward methods. Furthermore, we conduct an in-depth study that identifies critical implementation details and establishes well-justified standard practices in intrinsically-motivated RL. Our documentation, examples, and source code are available at https://github.com/RLE-Foundation/RLeXplore.