Benchmarking Partial Observability in Reinforcement Learning with a Suite of Memory-Improvable Domains

TL;DR

This paper introduces a comprehensive benchmark suite called POBAX for evaluating reinforcement learning algorithms under various forms of partial observability, emphasizing environments that require memory and are representative of real-world challenges.

Contribution

The paper provides guidelines for benchmarking partial observability, introduces the POBAX library with diverse environments, and demonstrates their memory-improvable nature for robust RL evaluation.

Findings

Environments in POBAX are all memory improvable.

The benchmark covers diverse partial observability scenarios.

Recommended hyperparameters facilitate rapid evaluation.

Abstract

Mitigating partial observability is a necessary but challenging task for general reinforcement learning algorithms. To improve an algorithm's ability to mitigate partial observability, researchers need comprehensive benchmarks to gauge progress. Most algorithms tackling partial observability are only evaluated on benchmarks with simple forms of state aliasing, such as feature masking and Gaussian noise. Such benchmarks do not represent the many forms of partial observability seen in real domains, like visual occlusion or unknown opponent intent. We argue that a partially observable benchmark should have two key properties. The first is coverage in its forms of partial observability, to ensure an algorithm's generalizability. The second is a large gap between the performance of a agents with more or less state information, all other factors roughly equal. This gap implies that an environment is memory improvable: where performance gains in a domain are from an algorithm's ability to cope with partial observability as opposed to other factors. We introduce best-practice guidelines for empirically benchmarking reinforcement learning under partial observability, as well as the open-source library POBAX: Partially Observable Benchmarks in JAX. We characterize the types of partial observability present in various environments and select representative environments for our benchmark. These environments include localization and mapping, visual control, games, and more. Additionally, we show that these tasks are all memory improvable and require hard-to-learn memory functions, providing a concrete signal for partial observability research. This framework includes recommended hyperparameters as well as algorithm implementations for fast, out-of-the-box evaluation, as well as highly performant environments implemented in JAX for GPU-scalable experimentation.