Inverse Concave-Utility Reinforcement Learning is Inverse Game Theory

TL;DR

This paper introduces a novel theoretical framework for inverse reinforcement learning with concave utilities, linking it to inverse game theory and mean-field games, addressing gaps in existing IRL methods.

Contribution

It develops a new approach to inverse CURL problems by establishing their equivalence to inverse game theory within mean-field games, which was not previously explored.

Findings

Most standard IRL results do not apply to CURL due to Bellman equation invalidation.

Proposes a new definition for feasible rewards in inverse CURL based on mean-field game equivalence.

Outlines future research directions and applications in human-AI collaboration.

Abstract

We consider inverse reinforcement learning problems with concave utilities. Concave Utility Reinforcement Learning (CURL) is a generalisation of the standard RL objective, which employs a concave function of the state occupancy measure, rather than a linear function. CURL has garnered recent attention for its ability to represent instances of many important applications including the standard RL such as imitation learning, pure exploration, constrained MDPs, offline RL, human-regularized RL, and others. Inverse reinforcement learning is a powerful paradigm that focuses on recovering an unknown reward function that can rationalize the observed behaviour of an agent. There has been recent theoretical advances in inverse RL where the problem is formulated as identifying the set of feasible reward functions. However, inverse RL for CURL problems has not been considered previously. In this paper we show that most of the standard IRL results do not apply to CURL in general, since CURL invalidates the classical Bellman equations. This calls for a new theoretical framework for the inverse CURL problem. Using a recent equivalence result between CURL and Mean-field Games, we propose a new definition for the feasible rewards for I-CURL by proving that this problem is equivalent to an inverse game theory problem in a subclass of mean-field games. We outline future directions and applications in human--AI collaboration enabled by our results.