Provably Efficient Off-Policy Adversarial Imitation Learning with Convergence Guarantees

TL;DR

This paper provides the first theoretical guarantees for off-policy Adversarial Imitation Learning, demonstrating its convergence and sample efficiency by reusing recent policy samples without importance sampling corrections.

Contribution

It establishes convergence guarantees and analyzes sample complexity for off-policy AIL algorithms, showing their theoretical soundness and efficiency.

Findings

Reusing recent policy samples maintains convergence guarantees.

Off-policy updates' distribution shift error is outweighed by increased data benefits.

First theoretical analysis of off-policy AIL algorithms.

Abstract

Adversarial Imitation Learning (AIL) faces challenges with sample inefficiency because of its reliance on sufficient on-policy data to evaluate the performance of the current policy during reward function updates. In this work, we study the convergence properties and sample complexity of off-policy AIL algorithms. We show that, even in the absence of importance sampling correction, reusing samples generated by the $o(\sqrt{K})$ most recent policies, where $K$ is the number of iterations of policy updates and reward updates, does not undermine the convergence guarantees of this class of algorithms. Furthermore, our results indicate that the distribution shift error induced by off-policy updates is dominated by the benefits of having more data available. This result provides theoretical support for the sample efficiency of off-policy AIL algorithms. To the best of our knowledge, this is the first work that provides theoretical guarantees for off-policy AIL algorithms.