Walking the Values in Bayesian Inverse Reinforcement Learning

TL;DR

This paper introduces ValueWalk, a novel Bayesian IRL method that efficiently samples in Q-value space rather than reward space, reducing computational costs and enabling better posterior estimation from demonstrations.

Contribution

The paper proposes a new sampling approach in Bayesian IRL focusing on Q-values, improving efficiency and scalability over traditional reward-based methods.

Findings

ValueWalk reduces computational costs in Bayesian IRL.

It enables efficient posterior sampling using Hamiltonian Monte Carlo.

Demonstrates improved performance on several benchmark tasks.

Abstract

The goal of Bayesian inverse reinforcement learning (IRL) is recovering a posterior distribution over reward functions using a set of demonstrations from an expert optimizing for a reward unknown to the learner. The resulting posterior over rewards can then be used to synthesize an apprentice policy that performs well on the same or a similar task. A key challenge in Bayesian IRL is bridging the computational gap between the hypothesis space of possible rewards and the likelihood, often defined in terms of Q values: vanilla Bayesian IRL needs to solve the costly forward planning problem - going from rewards to the Q values - at every step of the algorithm, which may need to be done thousands of times. We propose to solve this by a simple change: instead of focusing on primarily sampling in the space of rewards, we can focus on primarily working in the space of Q-values, since the computation required to go from Q-values to reward is radically cheaper. Furthermore, this reversion of the computation makes it easy to compute the gradient allowing efficient sampling using Hamiltonian Monte Carlo. We propose ValueWalk - a new Markov chain Monte Carlo method based on this insight - and illustrate its advantages on several tasks.