Inverse Q-Learning Done Right: Offline Imitation Learning in $Q^\pi$-Realizable MDPs

TL;DR

This paper introduces a new offline imitation learning algorithm, POIL, for linear and nonlinear Q-realizable MDPs, with theoretical guarantees and empirical success on benchmarks.

Contribution

It proposes POIL, a novel saddle-point based algorithm for offline imitation learning in Q-realizable MDPs, with proven performance guarantees and practical neural network implementation.

Findings

POIL matches expert performance with sample complexity in linear cases.

The method extends to nonlinear Q-realizable MDPs with higher sample complexity.

Neural POIL outperforms behavior cloning and rivals state-of-the-art algorithms.

Abstract

We study the problem of offline imitation learning in Markov decision processes (MDPs), where the goal is to learn a well-performing policy given a dataset of state-action pairs generated by an expert policy. Complementing a recent line of work on this topic that assumes the expert belongs to a tractable class of known policies, we approach this problem from a new angle and leverage a different type of structural assumption about the environment. Specifically, for the class of linear $Q^\pi$-realizable MDPs, we introduce a new algorithm called saddle-point offline imitation learning (\SPOIL), which is guaranteed to match the performance of any expert up to an additive error $\varepsilon$ with access to $\mathcal{O}(\varepsilon^{-2})$ samples. Moreover, we extend this result to possibly nonlinear $Q^\pi$-realizable MDPs at the cost of a worse sample complexity of order $\mathcal{O}(\varepsilon^{-4})$. Finally, our analysis suggests a new loss function for training critic networks from expert data in deep imitation learning. Empirical evaluations on standard benchmarks demonstrate that the neural net implementation of \SPOIL is superior to behavior cloning and competitive with state-of-the-art algorithms.