Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning

TL;DR

This paper investigates the transferability of rewards learned via inverse reinforcement learning, proposing new conditions and algorithms to ensure reward transferability across different transition laws using expert demonstrations.

Contribution

It introduces principal angles as a refined measure for transferability, and establishes new sufficient conditions and a PAC algorithm for learning transferable rewards from demonstrations.

Findings

Principal angles effectively measure similarity between transition laws.

Sufficient conditions for transferability from multiple experts with different transition laws.

A PAC algorithm for learning transferable rewards from expert demonstrations.

Abstract

Inverse reinforcement learning (IRL) aims to infer a reward from expert demonstrations, motivated by the idea that the reward, rather than the policy, is the most succinct and transferable description of a task [Ng et al., 2000]. However, the reward corresponding to an optimal policy is not unique, making it unclear if an IRL-learned reward is transferable to new transition laws in the sense that its optimal policy aligns with the optimal policy corresponding to the expert's true reward. Past work has addressed this problem only under the assumption of full access to the expert's policy, guaranteeing transferability when learning from two experts with the same reward but different transition laws that satisfy a specific rank condition [Rolland et al., 2022]. In this work, we show that the conditions developed under full access to the expert's policy cannot guarantee transferability in the more practical scenario where we have access only to demonstrations of the expert. Instead of a binary rank condition, we propose principal angles as a more refined measure of similarity and dissimilarity between transition laws. Based on this, we then establish two key results: 1) a sufficient condition for transferability to any transition laws when learning from at least two experts with sufficiently different transition laws, and 2) a sufficient condition for transferability to local changes in the transition law when learning from a single expert. Furthermore, we also provide a probably approximately correct (PAC) algorithm and an end-to-end analysis for learning transferable rewards from demonstrations of multiple experts.