Partial Identifiability in Inverse Reinforcement Learning For Agents With Non-Exponential Discounting

TL;DR

This paper investigates the limitations of inverse reinforcement learning in accurately inferring preferences of agents with non-exponential discounting, revealing fundamental challenges in preference identification for more realistic human-like decision models.

Contribution

It extends the theoretical understanding of IRL's partial identifiability to agents with non-exponential discounting, including hyperbolic discounting, highlighting new limitations.

Findings

IRL cannot fully identify preferences for non-exponentially discounted agents

Partial identifiability results apply broadly to hyperbolic and other non-exponential discounting models

IRL alone is often insufficient to determine the true reward function for such agents

Abstract

The aim of inverse reinforcement learning (IRL) is to infer an agent's preferences from observing their behaviour. Usually, preferences are modelled as a reward function, $R$, and behaviour is modelled as a policy, $\pi$. One of the central difficulties in IRL is that multiple preferences may lead to the same observed behaviour. That is, $R$ is typically underdetermined by $\pi$, which means that $R$ is only partially identifiable. Recent work has characterised the extent of this partial identifiability for different types of agents, including optimal and Boltzmann-rational agents. However, work so far has only considered agents that discount future reward exponentially: this is a serious limitation, especially given that extensive work in the behavioural sciences suggests that humans are better modelled as discounting hyperbolically. In this work, we newly characterise partial identifiability in IRL for agents with non-exponential discounting: our results are in particular relevant for hyperbolical discounting, but they also more generally apply to agents that use other types of (non-exponential) discounting. We significantly show that generally IRL is unable to infer enough information about $R$ to identify the correct optimal policy, which entails that IRL alone can be insufficient to adequately characterise the preferences of such agents.