Deep PQR: Solving Inverse Reinforcement Learning using Anchor Actions

TL;DR

This paper introduces PQR, a deep learning framework for inverse reinforcement learning that estimates reward functions considering action dependencies and stochastic transitions, using anchor actions for improved accuracy.

Contribution

The paper presents a novel deep energy-based IRL method called PQR that estimates policy, Q-function, and reward simultaneously, accommodating action-dependent rewards and stochastic dynamics.

Findings

PQR accurately recovers true rewards when environment transitions are known.

The method provides bounds on reward estimation error with unknown transitions.

Empirical results demonstrate PQR's effectiveness on synthetic and real datasets.

Abstract

We propose a reward function estimation framework for inverse reinforcement learning with deep energy-based policies. We name our method PQR, as it sequentially estimates the Policy, the $Q$-function, and the Reward function by deep learning. PQR does not assume that the reward solely depends on the state, instead it allows for a dependency on the choice of action. Moreover, PQR allows for stochastic state transitions. To accomplish this, we assume the existence of one anchor action whose reward is known, typically the action of doing nothing, yielding no reward. We present both estimators and algorithms for the PQR method. When the environment transition is known, we prove that the PQR reward estimator uniquely recovers the true reward. With unknown transitions, we bound the estimation error of PQR. Finally, the performance of PQR is demonstrated by synthetic and real-world datasets.