PufferLib: Making Reinforcement Learning Libraries and Environments Play Nice

TL;DR

PufferLib simplifies integrating reinforcement learning environments and libraries by providing compatibility wrappers and fast vectorization, enabling scalable training across diverse benchmarks and complex simulators.

Contribution

It introduces a library that ensures compatibility and accelerates training in reinforcement learning setups, supporting a wide range of environments and libraries.

Findings

Enables seamless use of popular RL libraries with various environments.

Provides fast vectorization for accelerated training.

Supports numerous benchmarks and complex simulators.

Abstract

You have an environment, a model, and a reinforcement learning library that are designed to work together but don't. PufferLib makes them play nice. The library provides one-line environment wrappers that eliminate common compatibility problems and fast vectorization to accelerate training. With PufferLib, you can use familiar libraries like CleanRL and SB3 to scale from classic benchmarks like Atari and Procgen to complex simulators like NetHack and Neural MMO. We release pip packages and prebuilt images with dependencies for dozens of environments. All of our code is free and open-source software under the MIT license, complete with baselines, documentation, and support at pufferai.github.io.