Optimistically Optimistic Exploration for Provably Efficient Infinite-Horizon Reinforcement and Imitation Learning

TL;DR

This paper introduces a new efficient algorithm for infinite-horizon linear MDPs that combines optimistic exploration techniques, achieving optimal regret bounds and advancing imitation learning with state-of-the-art results.

Contribution

The paper presents the first computationally efficient, rate-optimal algorithm for infinite-horizon linear MDPs using a novel combination of exploration bonuses and artificial transitions.

Findings

Achieves regret of order $ ilde{O}( ext{d}^3 (1- ext{γ})^{-7/2} T)$

Works against adversarial reward sequences

Sets new state-of-the-art in imitation learning for linear MDPs

Abstract

We study the problem of reinforcement learning in infinite-horizon discounted linear Markov decision processes (MDPs), and propose the first computationally efficient algorithm achieving rate-optimal regret guarantees in this setting. Our main idea is to combine two classic techniques for optimistic exploration: additive exploration bonuses applied to the reward function, and artificial transitions made to an absorbing state with maximal return. We show that, combined with a regularized approximate dynamic-programming scheme, the resulting algorithm achieves a regret of order $\tilde{\mathcal{O}} (\sqrt{d^3 (1 - \gamma)^{- 7 / 2} T})$, where $T$ is the total number of sample transitions, $\gamma \in (0,1)$ is the discount factor, and $d$ is the feature dimensionality. The results continue to hold against adversarial reward sequences, enabling application of our method to the problem of imitation learning in linear MDPs, where we achieve state-of-the-art results.