Receding Horizon Inverse Reinforcement Learning

TL;DR

Receding Horizon IRL (RHIRL) is a scalable and robust algorithm for inferring cost functions in high-dimensional, noisy, continuous systems by locally matching trajectories and stitching solutions, outperforming previous methods.

Contribution

Introduces RHIRL, a novel IRL algorithm that addresses scalability and robustness by local trajectory matching and cost function disentanglement in high-dimensional systems.

Findings

RHIRL outperforms existing IRL algorithms on benchmark tasks.

The cumulative error of RHIRL grows linearly with task duration.

RHIRL effectively handles noisy and imperfect expert demonstrations.

Abstract

Inverse reinforcement learning (IRL) seeks to infer a cost function that explains the underlying goals and preferences of expert demonstrations. This paper presents receding horizon inverse reinforcement learning (RHIRL), a new IRL algorithm for high-dimensional, noisy, continuous systems with black-box dynamic models. RHIRL addresses two key challenges of IRL: scalability and robustness. To handle high-dimensional continuous systems, RHIRL matches the induced optimal trajectories with expert demonstrations locally in a receding horizon manner and 'stitches' together the local solutions to learn the cost; it thereby avoids the 'curse of dimensionality'. This contrasts sharply with earlier algorithms that match with expert demonstrations globally over the entire high-dimensional state space. To be robust against imperfect expert demonstrations and control noise, RHIRL learns a state-dependent cost function 'disentangled' from system dynamics under mild conditions. Experiments on benchmark tasks show that RHIRL outperforms several leading IRL algorithms in most instances. We also prove that the cumulative error of RHIRL grows linearly with the task duration.