Learning Robust Rewards with Adversarial Inverse Reinforcement Learning

TL;DR

This paper introduces AIRL, a scalable adversarial inverse reinforcement learning algorithm that learns robust reward functions, enabling effective policy transfer across environments with varying dynamics.

Contribution

AIRL is a novel adversarial IRL method that recovers reward functions robust to environmental changes, improving transferability in high-dimensional problems.

Findings

AIRL outperforms prior IRL methods in transfer scenarios.

AIRL learns reward functions robust to dynamics variations.

AIRL enables policy learning in high-dimensional, changing environments.

Abstract

Reinforcement learning provides a powerful and general framework for decision making and control, but its application in practice is often hindered by the need for extensive feature and reward engineering. Deep reinforcement learning methods can remove the need for explicit engineering of policy or value features, but still require a manually specified reward function. Inverse reinforcement learning holds the promise of automatic reward acquisition, but has proven exceptionally difficult to apply to large, high-dimensional problems with unknown dynamics. In this work, we propose adverserial inverse reinforcement learning (AIRL), a practical and scalable inverse reinforcement learning algorithm based on an adversarial reward learning formulation. We demonstrate that AIRL is able to recover reward functions that are robust to changes in dynamics, enabling us to learn policies even under significant variation in the environment seen during training. Our experiments show that AIRL greatly outperforms prior methods in these transfer settings.