A Bayesian Solution To The Imitation Gap

TL;DR

This paper introduces BIG, a Bayesian approach that uses inverse reinforcement learning to address the imitation gap caused by observability differences, enabling agents to explore effectively and learn optimal policies from demonstrations.

Contribution

The paper presents a Bayesian method that infers reward functions to overcome the imitation gap, allowing agents to explore and adapt in environments with partial observability.

Findings

BIG enables exploration at test time in the presence of an imitation gap.

BIG outperforms naive imitation learning in environments with observability differences.

The approach effectively learns optimal policies using expert demonstrations despite the imitation gap.

Abstract

In many real-world settings, an agent must learn to act in environments where no reward signal can be specified, but a set of expert demonstrations is available. Imitation learning (IL) is a popular framework for learning policies from such demonstrations. However, in some cases, differences in observability between the expert and the agent can give rise to an imitation gap such that the expert's policy is not optimal for the agent and a naive application of IL can fail catastrophically. In particular, if the expert observes the Markov state and the agent does not, then the expert will not demonstrate the information-gathering behavior needed by the agent but not the expert. In this paper, we propose a Bayesian solution to the Imitation Gap (BIG), first using the expert demonstrations, together with a prior specifying the cost of exploratory behavior that is not demonstrated, to infer a posterior over rewards with Bayesian inverse reinforcement learning (IRL). BIG then uses the reward posterior to learn a Bayes-optimal policy. Our experiments show that BIG, unlike IL, allows the agent to explore at test time when presented with an imitation gap, whilst still learning to behave optimally using expert demonstrations when no such gap exists.