Imitation Learning in Discounted Linear MDPs without exploration assumptions

TL;DR

This paper introduces ILARL, a new imitation learning algorithm for infinite horizon linear MDPs that removes exploration assumptions and improves sample complexity bounds, supported by theoretical analysis and numerical experiments.

Contribution

The paper proposes ILARL, a novel imitation learning algorithm that removes exploration assumptions and improves sample complexity bounds for linear MDPs.

Findings

ILARL outperforms existing algorithms in numerical experiments.

Achieves improved sample complexity bounds of O(ε^{-4}) for infinite horizon case.

Provides the first results for online learning in adversarial loss settings for infinite horizon linear MDPs.

Abstract

We present a new algorithm for imitation learning in infinite horizon linear MDPs dubbed ILARL which greatly improves the bound on the number of trajectories that the learner needs to sample from the environment. In particular, we remove exploration assumptions required in previous works and we improve the dependence on the desired accuracy $\epsilon$ from $\mathcal{O}(\epsilon^{-5})$ to $\mathcal{O}(\epsilon^{-4})$. Our result relies on a connection between imitation learning and online learning in MDPs with adversarial losses. For the latter setting, we present the first result for infinite horizon linear MDP which may be of independent interest. Moreover, we are able to provide a strengthen result for the finite horizon case where we achieve $\mathcal{O}(\epsilon^{-2})$. Numerical experiments with linear function approximation shows that ILARL outperforms other commonly used algorithms.