PAC Apprenticeship Learning with Bayesian Active Inverse Reinforcement Learning

TL;DR

This paper introduces PAC-EIG, a novel information-theoretic method for active inverse reinforcement learning that guarantees probably-approximately-correct policies with fewer demonstrations, especially in safety-critical domains.

Contribution

It provides the first PAC guarantee for active IRL with noisy demonstrations and introduces a new acquisition function that maximizes information gain about policy regret.

Findings

PAC-EIG achieves reliable policies with fewer demonstrations.

Theoretical convergence bounds are established for finite state-action spaces.

Experimental results demonstrate advantages over prior heuristic methods.

Abstract

As AI systems become increasingly autonomous, reliably aligning their decision-making with human preferences is essential. Inverse reinforcement learning (IRL) offers a promising approach to infer preferences from demonstrations. These preferences can then be used to produce an apprentice policy that performs well on the demonstrated task. However, in domains like autonomous driving or robotics, where errors can have serious consequences, we need not just good average performance but reliable policies with formal guarantees -- yet obtaining sufficient human demonstrations for reliability guarantees can be costly. Active IRL addresses this challenge by strategically selecting the most informative scenarios for human demonstration. We introduce PAC-EIG, an information-theoretic acquisition function that directly targets probably-approximately-correct (PAC) guarantees for the learned policy -- providing the first such theoretical guarantee for active IRL with noisy expert demonstrations. Our method maximises information gain about the regret of the apprentice policy, efficiently identifying states requiring further demonstration. We also present Reward-EIG as an alternative when learning the reward itself is the primary objective. Focusing on finite state-action spaces, we prove convergence bounds, illustrate failure modes of prior heuristic methods, and demonstrate our method's advantages experimentally.