Demonstration-efficient Inverse Reinforcement Learning in Procedurally Generated Environments

TL;DR

This paper introduces DE-AIRL, a demonstration-efficient inverse reinforcement learning method tailored for procedurally generated environments, enabling reward extrapolation with fewer expert demonstrations and improved generalization.

Contribution

The paper presents DE-AIRL, a novel adversarial IRL approach that reduces demonstration requirements and enhances reward learning in procedurally generated domains.

Findings

DE-AIRL significantly decreases the number of demonstrations needed.

The method generalizes reward functions across procedural environments.

Effective on MiniGrid and DeepCrawl benchmarks.

Abstract

Deep Reinforcement Learning achieves very good results in domains where reward functions can be manually engineered. At the same time, there is growing interest within the community in using games based on Procedurally Content Generation (PCG) as benchmark environments since this type of environment is perfect for studying overfitting and generalization of agents under domain shift. Inverse Reinforcement Learning (IRL) can instead extrapolate reward functions from expert demonstrations, with good results even on high-dimensional problems, however there are no examples of applying these techniques to procedurally-generated environments. This is mostly due to the number of demonstrations needed to find a good reward model. We propose a technique based on Adversarial Inverse Reinforcement Learning which can significantly decrease the need for expert demonstrations in PCG games. Through the use of an environment with a limited set of initial seed levels, plus some modifications to stabilize training, we show that our approach, DE-AIRL, is demonstration-efficient and still able to extrapolate reward functions which generalize to the fully procedural domain. We demonstrate the effectiveness of our technique on two procedural environments, MiniGrid and DeepCrawl, for a variety of tasks.