How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach

TL;DR

This paper introduces a new, scalable approach to online Inverse Reinforcement Learning in large state spaces, overcoming previous limitations by developing the CATY-IRL algorithm with complexity independent of state space size.

Contribution

The paper proposes the rewards compatibility framework and the CATY-IRL algorithm, achieving sample efficiency and minimax optimality in large Linear MDPs, and unifying IRL and RFE.

Findings

CATY-IRL's complexity is independent of state space size.

CATY-IRL is minimax optimal up to logarithmic factors in tabular settings.

Reward-Free Exploration rate matches worst-case bounds, improving previous results.

Abstract

In online Inverse Reinforcement Learning (IRL), the learner can collect samples about the dynamics of the environment to improve its estimate of the reward function. Since IRL suffers from identifiability issues, many theoretical works on online IRL focus on estimating the entire set of rewards that explain the demonstrations, named the feasible reward set. However, none of the algorithms available in the literature can scale to problems with large state spaces. In this paper, we focus on the online IRL problem in Linear Markov Decision Processes (MDPs). We show that the structure offered by Linear MDPs is not sufficient for efficiently estimating the feasible set when the state space is large. As a consequence, we introduce the novel framework of rewards compatibility, which generalizes the notion of feasible set, and we develop CATY-IRL, a sample efficient algorithm whose complexity is independent of the cardinality of the state space in Linear MDPs. When restricted to the tabular setting, we demonstrate that CATY-IRL is minimax optimal up to logarithmic factors. As a by-product, we show that Reward-Free Exploration (RFE) enjoys the same worst-case rate, improving over the state-of-the-art lower bound. Finally, we devise a unifying framework for IRL and RFE that may be of independent interest.