NS-Gym: Open-Source Simulation Environments and Benchmarks for Non-Stationary Markov Decision Processes

TL;DR

NS-Gym is an open-source simulation toolkit designed for non-stationary Markov decision processes, providing standardized environments and benchmarks to facilitate research on decision-making in changing conditions.

Contribution

It introduces NS-Gym, the first modular simulation environment with standardized benchmarks for evaluating algorithms in non-stationary MDPs.

Findings

Benchmarking six algorithms demonstrates varied adaptability to non-stationarity.

NS-Gym enables reproducible evaluation of decision-making algorithms.

The toolkit facilitates systematic study of non-stationary environments.

Abstract

In many real-world applications, agents must make sequential decisions in environments where conditions are subject to change due to various exogenous factors. These non-stationary environments pose significant challenges to traditional decision-making models, which typically assume stationary dynamics. Non-stationary Markov decision processes (NS-MDPs) offer a framework to model and solve decision problems under such changing conditions. However, the lack of standardized benchmarks and simulation tools has hindered systematic evaluation and advance in this field. We present NS-Gym, the first simulation toolkit designed explicitly for NS-MDPs, integrated within the popular Gymnasium framework. In NS-Gym, we segregate the evolution of the environmental parameters that characterize non-stationarity from the agent's decision-making module, allowing for modular and flexible adaptations to dynamic environments. We review prior work in this domain and present a toolkit encapsulating key problem characteristics and types in NS-MDPs. This toolkit is the first effort to develop a set of standardized interfaces and benchmark problems to enable consistent and reproducible evaluation of algorithms under non-stationary conditions. We also benchmark six algorithmic approaches from prior work on NS-MDPs using NS-Gym. Our vision is that NS-Gym will enable researchers to assess the adaptability and robustness of their decision-making algorithms to non-stationary conditions.