On the Effective Horizon of Inverse Reinforcement Learning

TL;DR

This paper analyzes how the effective time horizon in inverse reinforcement learning influences reward estimation accuracy and overfitting, proposing a joint reward and horizon learning approach validated by experiments.

Contribution

It provides a formal analysis of the effective horizon phenomenon in IRL and introduces a joint reward and horizon learning framework.

Findings

Shorter effective horizons can improve IRL performance.

Joint reward and horizon learning outperforms traditional methods.

Experimental results support the theoretical analysis.

Abstract

Inverse reinforcement learning (IRL) algorithms often rely on (forward) reinforcement learning or planning, over a given time horizon, to compute an approximately optimal policy for a hypothesized reward function; they then match this policy with expert demonstrations. The time horizon plays a critical role in determining both the accuracy of reward estimates and the computational efficiency of IRL algorithms. Interestingly, an *effective time horizon* shorter than the ground-truth value often produces better results faster. This work formally analyzes this phenomenon and provides an explanation: the time horizon controls the complexity of an induced policy class and mitigates overfitting with limited data. This analysis provides a guide for the principled choice of the effective horizon for IRL. It also prompts us to re-examine the classic IRL formulation: it is more natural to learn jointly the reward and the effective horizon rather than the reward alone with a given horizon. To validate our findings, we implement a cross-validation extension and the experimental results support the theoretical analysis. The project page and code are publicly available.