Inverse Reinforcement Learning via Matching of Optimality Profiles

TL;DR

This paper introduces a novel inverse reinforcement learning method that leverages suboptimal and heterogeneous demonstrations by matching optimality profiles, enabling reward inference with limited optimal data.

Contribution

It proposes a reward learning algorithm that uses optimality profiles and Wasserstein distance minimization, effectively handling suboptimal demonstrations.

Findings

The method successfully learns reward functions from suboptimal demonstrations.

Policies trained on these rewards outperform the original demonstrations.

The approach accommodates weak supervision signals like reward distributions.

Abstract

The goal of inverse reinforcement learning (IRL) is to infer a reward function that explains the behavior of an agent performing a task. The assumption that most approaches make is that the demonstrated behavior is near-optimal. In many real-world scenarios, however, examples of truly optimal behavior are scarce, and it is desirable to effectively leverage sets of demonstrations of suboptimal or heterogeneous performance, which are easier to obtain. We propose an algorithm that learns a reward function from such demonstrations together with a weak supervision signal in the form of a distribution over rewards collected during the demonstrations (or, more generally, a distribution over cumulative discounted future rewards). We view such distributions, which we also refer to as optimality profiles, as summaries of the degree of optimality of the demonstrations that may, for example, reflect the opinion of a human expert. Given an optimality profile and a small amount of additional supervision, our algorithm fits a reward function, modeled as a neural network, by essentially minimizing the Wasserstein distance between the corresponding induced distribution and the optimality profile. We show that our method is capable of learning reward functions such that policies trained to optimize them outperform the demonstrations used for fitting the reward functions.