A Framework and Method for Online Inverse Reinforcement Learning

TL;DR

This paper introduces a formal framework and a novel method for online inverse reinforcement learning (IRL), enabling incremental learning of agent preferences with performance guarantees, suitable for real-time applications.

Contribution

It presents the first formal framework for online IRL and extends maximum entropy IRL to this setting with proven performance improvements and error bounds.

Findings

Method shows monotonic performance improvement with more data

Error bounds are probabilistically established under various observability conditions

Experiments demonstrate improved speed and effectiveness in robotic tasks

Abstract

Inverse reinforcement learning (IRL) is the problem of learning the preferences of an agent from the observations of its behavior on a task. While this problem has been well investigated, the related problem of {\em online} IRL---where the observations are incrementally accrued, yet the demands of the application often prohibit a full rerun of an IRL method---has received relatively less attention. We introduce the first formal framework for online IRL, called incremental IRL (I2RL), and a new method that advances maximum entropy IRL with hidden variables, to this setting. Our formal analysis shows that the new method has a monotonically improving performance with more demonstration data, as well as probabilistically bounded error, both under full and partial observability. Experiments in a simulated robotic application of penetrating a continuous patrol under occlusion shows the relatively improved performance and speed up of the new method and validates the utility of online IRL.