Environment Design for Inverse Reinforcement Learning

TL;DR

This paper proposes an adaptive environment design framework for inverse reinforcement learning that enhances sample-efficiency and robustness by selecting environments to quickly infer reward functions from expert demonstrations.

Contribution

It introduces a novel adaptive environment selection method that improves the efficiency and robustness of inverse reinforcement learning.

Findings

Significant reduction in sample complexity.

Enhanced robustness to environment dynamics changes.

Effective in both exact and approximate inference scenarios.

Abstract

Learning a reward function from demonstrations suffers from low sample-efficiency. Even with abundant data, current inverse reinforcement learning methods that focus on learning from a single environment can fail to handle slight changes in the environment dynamics. We tackle these challenges through adaptive environment design. In our framework, the learner repeatedly interacts with the expert, with the former selecting environments to identify the reward function as quickly as possible from the expert's demonstrations in said environments. This results in improvements in both sample-efficiency and robustness, as we show experimentally, for both exact and approximate inference.