Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platform

TL;DR

Cleanba is an open-source platform for distributed deep reinforcement learning that enhances reproducibility, reduces training time, and maintains high performance across different hardware setups.

Contribution

The paper introduces Cleanba, a highly reproducible and efficient distributed DRL platform with optimized implementations of PPO and IMPALA.

Findings

Cleanba achieves comparable or higher scores than existing baselines.

Cleanba reduces training time compared to traditional frameworks.

Cleanba provides more consistent learning curves across hardware setups.

Abstract

Distributed Deep Reinforcement Learning (DRL) aims to leverage more computational resources to train autonomous agents with less training time. Despite recent progress in the field, reproducibility issues have not been sufficiently explored. This paper first shows that the typical actor-learner framework can have reproducibility issues even if hyperparameters are controlled. We then introduce Cleanba, a new open-source platform for distributed DRL that proposes a highly reproducible architecture. Cleanba implements highly optimized distributed variants of PPO and IMPALA. Our Atari experiments show that these variants can obtain equivalent or higher scores than strong IMPALA baselines in moolib and torchbeast and PPO baseline in CleanRL. However, Cleanba variants present 1) shorter training time and 2) more reproducible learning curves in different hardware settings. Cleanba's source code is available at \url{https://github.com/vwxyzjn/cleanba}