Repeated Inverse Reinforcement Learning

TL;DR

This paper introduces a new repeated inverse reinforcement learning problem where an agent learns to act on behalf of a human across multiple tasks, aiming to minimize surprising the human and efficiently updating its behavior with demonstrations.

Contribution

It formalizes the repeated IRL problem, explores different task sequence models, and provides foundational theoretical results for this novel setting.

Findings

Formalization of the repeated IRL problem

Analysis of task sequence selection methods

Foundational theoretical results for the problem

Abstract

We introduce a novel repeated Inverse Reinforcement Learning problem: the agent has to act on behalf of a human in a sequence of tasks and wishes to minimize the number of tasks that it surprises the human by acting suboptimally with respect to how the human would have acted. Each time the human is surprised, the agent is provided a demonstration of the desired behavior by the human. We formalize this problem, including how the sequence of tasks is chosen, in a few different ways and provide some foundational results.