Lamarckian Platform: Pushing the Boundaries of Evolutionary Reinforcement Learning towards Asynchronous Commercial Games

TL;DR

Lamarckian is an open-source platform that enhances evolutionary reinforcement learning for asynchronous commercial games by enabling scalable, efficient, and asynchronous training workflows on distributed computing resources.

Contribution

The paper introduces Lamarckian, a novel platform supporting scalable, asynchronous evolutionary reinforcement learning with optimized communication and modular architecture for commercial game applications.

Findings

Training speed doubled on Google football with PPO.

Training speed increased 13-fold on Pong with PBT+PPO.

Lamarckian effectively supports behavior diversity and game balancing in commercial games.

Abstract

Despite the emerging progress of integrating evolutionary computation into reinforcement learning, the absence of a high-performance platform endowing composability and massive parallelism causes non-trivial difficulties for research and applications related to asynchronous commercial games. Here we introduce Lamarckian - an open-source platform featuring support for evolutionary reinforcement learning scalable to distributed computing resources. To improve the training speed and data efficiency, Lamarckian adopts optimized communication methods and an asynchronous evolutionary reinforcement learning workflow. To meet the demand for an asynchronous interface by commercial games and various methods, Lamarckian tailors an asynchronous Markov Decision Process interface and designs an object-oriented software architecture with decoupled modules. In comparison with the state-of-the-art RLlib, we empirically demonstrate the unique advantages of Lamarckian on benchmark tests with up to 6000 CPU cores: i) both the sampling efficiency and training speed are doubled when running PPO on Google football game; ii) the training speed is 13 times faster when running PBT+PPO on Pong game. Moreover, we also present two use cases: i) how Lamarckian is applied to generating behavior-diverse game AI; ii) how Lamarckian is applied to game balancing tests for an asynchronous commercial game.