Provably and Practically Efficient Adversarial Imitation Learning with General Function Approximation

TL;DR

This paper introduces OPT-AIL, a new adversarial imitation learning method with general function approximation, providing theoretical guarantees and practical efficiency, and demonstrating superior empirical performance over existing methods.

Contribution

The paper presents the first provably efficient AIL method with general function approximation, combining theoretical analysis with practical optimization-based implementation.

Findings

OPT-AIL achieves polynomial sample complexity.

OPT-AIL outperforms previous deep AIL methods in challenging tasks.

The method simplifies practical implementation by requiring only two objective optimizations.

Abstract

As a prominent category of imitation learning methods, adversarial imitation learning (AIL) has garnered significant practical success powered by neural network approximation. However, existing theoretical studies on AIL are primarily limited to simplified scenarios such as tabular and linear function approximation and involve complex algorithmic designs that hinder practical implementation, highlighting a gap between theory and practice. In this paper, we explore the theoretical underpinnings of online AIL with general function approximation. We introduce a new method called optimization-based AIL (OPT-AIL), which centers on performing online optimization for reward functions and optimism-regularized Bellman error minimization for Q-value functions. Theoretically, we prove that OPT-AIL achieves polynomial expert sample complexity and interaction complexity for learning near-expert policies. To our best knowledge, OPT-AIL is the first provably efficient AIL method with general function approximation. Practically, OPT-AIL only requires the approximate optimization of two objectives, thereby facilitating practical implementation. Empirical studies demonstrate that OPT-AIL outperforms previous state-of-the-art deep AIL methods in several challenging tasks.