oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally Extended Actions

TL;DR

This paper introduces oIRL, a robust adversarial IRL method that learns hierarchical, disentangled rewards and policies over options, improving transferability and robustness in complex environments.

Contribution

The paper proposes a hierarchical adversarial IRL algorithm that learns disentangled rewards and options, enhancing generalization and robustness in transfer learning tasks.

Findings

Achieves comparable results to state-of-the-art in continuous control benchmarks.

Learns generalizable policies and reward functions in complex transfer tasks.

Reduces reward entanglement, improving robustness to environment changes.

Abstract

Explicit engineering of reward functions for given environments has been a major hindrance to reinforcement learning methods. While Inverse Reinforcement Learning (IRL) is a solution to recover reward functions from demonstrations only, these learned rewards are generally heavily \textit{entangled} with the dynamics of the environment and therefore not portable or \emph{robust} to changing environments. Modern adversarial methods have yielded some success in reducing reward entanglement in the IRL setting. In this work, we leverage one such method, Adversarial Inverse Reinforcement Learning (AIRL), to propose an algorithm that learns hierarchical disentangled rewards with a policy over options. We show that this method has the ability to learn \emph{generalizable} policies and reward functions in complex transfer learning tasks, while yielding results in continuous control benchmarks that are comparable to those of the state-of-the-art methods.