Rogue-Gym: A New Challenge for Generalization in Reinforcement Learning

TL;DR

Rogue-Gym is a roguelike environment designed to evaluate the generalization capabilities of reinforcement learning agents, revealing limitations of current methods and the need for improved generalization techniques.

Contribution

This paper introduces Rogue-Gym as a new benchmark for testing RL generalization, highlighting the challenges and limitations of existing methods like PPO.

Findings

Standard PPO struggles with overfitting in Rogue-Gym.

Some enhancements slightly improve generalization, but overall challenges remain.

Rogue-Gym provides a diverse and fair environment for assessing RL generalization.

Abstract

In this paper, we propose Rogue-Gym, a simple and classic style roguelike game built for evaluating generalization in reinforcement learning (RL). Combined with the recent progress of deep neural networks, RL has successfully trained human-level agents without human knowledge in many games such as those for Atari 2600. However, it has been pointed out that agents trained with RL methods often overfit the training environment, and they work poorly in slightly different environments. To investigate this problem, some research environments with procedural content generation have been proposed. Following these studies, we propose the use of roguelikes as a benchmark for evaluating the generalization ability of RL agents. In our Rogue-Gym, agents need to explore dungeons that are structured differently each time they start a new game. Thanks to the very diverse structures of the dungeons, we believe that the generalization benchmark of Rogue-Gym is sufficiently fair. In our experiments, we evaluate a standard reinforcement learning method, PPO, with and without enhancements for generalization. The results show that some enhancements believed to be effective fail to mitigate the overfitting in Rogue-Gym, although others slightly improve the generalization ability.