BC-IRL: Learning Generalizable Reward Functions from Demonstrations

TL;DR

This paper introduces BC-IRL, a novel inverse reinforcement learning method that learns reward functions with improved generalization capabilities, outperforming maximum-entropy IRL in various robotic tasks.

Contribution

BC-IRL is a new IRL approach that updates reward functions to better match expert policies, enhancing generalization beyond demonstration coverage.

Findings

BC-IRL achieves over twice the success rate of baselines in generalization tasks.

BC-IRL learns rewards that better generalize to unseen states.

The method outperforms maximum-entropy IRL in robotic control experiments.

Abstract

How well do reward functions learned with inverse reinforcement learning (IRL) generalize? We illustrate that state-of-the-art IRL algorithms, which maximize a maximum-entropy objective, learn rewards that overfit to the demonstrations. Such rewards struggle to provide meaningful rewards for states not covered by the demonstrations, a major detriment when using the reward to learn policies in new situations. We introduce BC-IRL a new inverse reinforcement learning method that learns reward functions that generalize better when compared to maximum-entropy IRL approaches. In contrast to the MaxEnt framework, which learns to maximize rewards around demonstrations, BC-IRL updates reward parameters such that the policy trained with the new reward matches the expert demonstrations better. We show that BC-IRL learns rewards that generalize better on an illustrative simple task and two continuous robotic control tasks, achieving over twice the success rate of baselines in challenging generalization settings.